Running Distributed VERONA Experiments with SLURM: A Complete Guide

VERONA separates experiment orchestration from verification workloads, enabling large-scale robustness analysis across SLURM-managed clusters using ExperimentRepository for context management and per-job workers that execute via sbatch.

The VERONA framework (from the ada-research/verona repository) is designed to scale neural network verification from single-machine prototypes to cluster-wide campaigns. By leveraging SLURM’s job scheduler, researchers can distribute thousands of robustness queries across GPU nodes while maintaining centralized result aggregation.

Core Architecture for Distributed Execution

VERONA’s distributed mode relies on three pluggable components that serialize verification tasks into self-contained units.

ExperimentRepository

The ExperimentRepository class in ada_verona/database/experiment_repository.py acts as the central authority for all file system paths. It provides create_verification_context to bundle a network, data point, and property into a single object, and save_verification_context_to_yaml to persist that context to disk so that remote workers can reconstruct the exact task.

# From ada_verona/database/experiment_repository.py

def create_verification_context(self, network, data_point, property_generator):
    # Bundles network, input, and robustness property

    ...

def save_verification_context_to_yaml(self, path: Path, verification_context):
    # Serializes context for remote workers

    ...

Property Generators

Distributed jobs require a formal robustness specification. The One2AnyPropertyGenerator in ada_verona/verification_module/property_generator/one2any_property_generator.py generates "one-to-any" robustness properties for a specified number of output classes and data bounds (e.g., [0, 1]).

Dataset Samplers

To avoid verifying every input, PredictionsBasedSampler (from ada_verona/dataset_sampler/predictions_based_sampler.py) selects specific data points—such as correctly classified images—so that each SLURM job focuses on meaningful verification queries.

Orchestrating SLURM Jobs

The master script main_create_robustness_dist_multiple_jobs.py (located in examples/scripts/multiple_jobs/) automates the creation and submission of hundreds or thousands of verification tasks.

Generating SLURM Scripts

The helper function write_slurm_script formats a Bash template with concrete paths, the epsilon list, and the Python worker command. It writes the final script to a temporary file that sbatch will later execute.

# From examples/scripts/multiple_jobs/main_create_robustness_dist_multiple_jobs.py

def write_slurm_script(
    slurm_script_template: str,
    slurm_scripts_path: Path,
    file_verification_context: Path,
    base_path_experiment_repository: Path,
    network_folder: Path,
    experiment_name: Path,
    epsilon_list: np.ndarray,
    temp_slurm_script: Path,
):
    epsilon_list_str = " ".join(map(str, epsilon_list))
    slurm_script_content = slurm_script_template.format(
        slurm_scripts_path=slurm_scripts_path,
        file_verification_context=file_verification_context,
        base_path_experiment_repository=base_path_experiment_repository,
        network_folder=network_folder,
        experiment_name=experiment_name,
        epsilon_list=epsilon_list_str,
    )
    with open(temp_slurm_script, "w") as f:
        f.write(slurm_script_content)

Submitting to the Cluster

Once the script is written, run_slurm_script makes it executable and invokes sbatch to queue the job on the SLURM-managed cluster.

# From examples/scripts/multiple_jobs/main_create_robustness_dist_multiple_jobs.py

def run_slurm_script(temp_slurm_script: Path):
    os.chmod(temp_slurm_script, stat.S_IRWXU)   # make executable

    os.system(f"sbatch {temp_slurm_script}")   # schedule on the cluster

The Verification Worker

Each SLURM node executes one_multiple_jobs.py, which acts as a stateless worker that reconstructs the verification context and runs the formal analysis.

Per-Job Execution

The worker receives paths and parameters via command-line arguments, reloads the ExperimentRepository, and deserializes the verification context from the YAML file created by the master script.

# From examples/scripts/multiple_jobs/one_multiple_jobs.py

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--file_verification_context", type=Path)
    parser.add_argument("--base_path_experiment_repository")
    parser.add_argument("--network_folder")
    parser.add_argument("--experiment_name")
    parser.add_argument("--epsilon_list", type=float, nargs="+")
    args = parser.parse_args()

    repo = ExperimentRepository(
        base_path=Path(args.base_path_experiment_repository),
        network_folder=Path(args.network_folder),
    )
    repo.load_experiment(experiment_name=args.experiment_name)

    verifier = AutoVerifyModule(verifier=AbCrown(), timeout=360)
    estimator = BinarySearchEpsilonValueEstimator(
        epsilon_value_list=args.epsilon_list.copy(),
        verifier=verifier,
    )
    vc = repo.load_verification_context_from_yaml(
        Path(args.file_verification_context)
    )
    result = estimator.compute_epsilon_value(vc)
    repo.save_result(result)

Aggregating Results and Reporting

After the cluster finishes, the ExperimentRepository contains a consolidated result_df.csv and optional per_epsilon_results.csv. The ReportCreator class in ada_verona/analysis/report_creator.py can then generate visualisations—such as histograms, box-plots, KDE, and ECDF plots—from these aggregated results to analyze the robustness distribution across the network population.

Summary

• VERONA decouples experiment orchestration from verification execution, making it ideal for distributed SLURM clusters.
• The ExperimentRepository in ada_verona/database/experiment_repository.py centralizes path management and context serialization via save_verification_context_to_yaml and load_verification_context_from_yaml.
• main_create_robustness_dist_multiple_jobs.py generates per-job SLURM scripts using write_slurm_script and submits them with run_slurm_script.
• Each node executes one_multiple_jobs.py, which reconstructs the verification context and runs BinarySearchEpsilonValueEstimator to compute robustness bounds.
• Results are automatically aggregated into CSV files and can be visualized using ReportCreator.

Frequently Asked Questions

How does VERONA handle file paths across distributed SLURM nodes?

VERONA uses the ExperimentRepository class to abstract all file system operations. When the master script creates a job, it serializes the verification context—including absolute paths—to a YAML file using save_verification_context_to_yaml. The worker node then calls load_verification_context_from_yaml to reconstruct the exact same paths, ensuring consistency across the cluster.

What verification backend does the SLURM worker use?

The example worker in one_multiple_jobs.py instantiates an AutoVerifyModule backed by the AutoCROWN verifier (AbCrown class) with a 360-second timeout. You can swap this for other verifiers supported by VERONA’s modular backend interface, provided they implement the same API used by BinarySearchEpsilonValueEstimator.

How do I customize the SLURM job template?

Modify the slurm_script_template string inside main_create_robustness_dist_multiple_jobs.py. The template uses Python’s str.format method to inject paths and the epsilon list. You can adjust #SBATCH directives—such as --partition, --exclude, or GPU requests—to match your cluster’s configuration before the jobs are written and submitted via sbatch.

Where are the results stored after the cluster run completes?

The ExperimentRepository automatically persists results to result_df.csv (aggregated robustness values) and optionally per_epsilon_results.csv (detailed per-epsilon verification outcomes) within the experiment directory. These CSV files can be loaded directly into pandas or processed by ReportCreator in ada_verona/analysis/report_creator.py to generate publication-ready histograms and ECDF plots.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}