# Running Distributed VERONA Experiments with SLURM: A Complete Guide

> Master running distributed VERONA experiments on SLURM clusters. This guide details using ExperimentRepository and sbatch for efficient, large-scale robustness analysis.

- Repository: [ADA research/verona](https://github.com/ada-research/verona)
- Tags: how-to-guide
- Published: 2026-02-23

---

**VERONA separates experiment orchestration from verification workloads, enabling large-scale robustness analysis across SLURM-managed clusters using `ExperimentRepository` for context management and per-job workers that execute via `sbatch`.**

The VERONA framework (from the `ada-research/verona` repository) is designed to scale neural network verification from single-machine prototypes to cluster-wide campaigns. By leveraging SLURM’s job scheduler, researchers can distribute thousands of robustness queries across GPU nodes while maintaining centralized result aggregation.

## Core Architecture for Distributed Execution

VERONA’s distributed mode relies on three pluggable components that serialize verification tasks into self-contained units.

### ExperimentRepository

The `ExperimentRepository` class in [`ada_verona/database/experiment_repository.py`](https://github.com/ada-research/verona/blob/main/ada_verona/database/experiment_repository.py) acts as the central authority for all file system paths. It provides `create_verification_context` to bundle a network, data point, and property into a single object, and `save_verification_context_to_yaml` to persist that context to disk so that remote workers can reconstruct the exact task.

```python

# From ada_verona/database/experiment_repository.py

def create_verification_context(self, network, data_point, property_generator):
    # Bundles network, input, and robustness property

    ...

def save_verification_context_to_yaml(self, path: Path, verification_context):
    # Serializes context for remote workers

    ...

```

### Property Generators

Distributed jobs require a formal robustness specification. The `One2AnyPropertyGenerator` in [`ada_verona/verification_module/property_generator/one2any_property_generator.py`](https://github.com/ada-research/verona/blob/main/ada_verona/verification_module/property_generator/one2any_property_generator.py) generates "one-to-any" robustness properties for a specified number of output classes and data bounds (e.g., `[0, 1]`).

### Dataset Samplers

To avoid verifying every input, `PredictionsBasedSampler` (from [`ada_verona/dataset_sampler/predictions_based_sampler.py`](https://github.com/ada-research/verona/blob/main/ada_verona/dataset_sampler/predictions_based_sampler.py)) selects specific data points—such as correctly classified images—so that each SLURM job focuses on meaningful verification queries.

## Orchestrating SLURM Jobs

The master script [`main_create_robustness_dist_multiple_jobs.py`](https://github.com/ada-research/verona/blob/main/main_create_robustness_dist_multiple_jobs.py) (located in `examples/scripts/multiple_jobs/`) automates the creation and submission of hundreds or thousands of verification tasks.

### Generating SLURM Scripts

The helper function `write_slurm_script` formats a Bash template with concrete paths, the epsilon list, and the Python worker command. It writes the final script to a temporary file that `sbatch` will later execute.

```python

# From examples/scripts/multiple_jobs/main_create_robustness_dist_multiple_jobs.py

def write_slurm_script(
    slurm_script_template: str,
    slurm_scripts_path: Path,
    file_verification_context: Path,
    base_path_experiment_repository: Path,
    network_folder: Path,
    experiment_name: Path,
    epsilon_list: np.ndarray,
    temp_slurm_script: Path,
):
    epsilon_list_str = " ".join(map(str, epsilon_list))
    slurm_script_content = slurm_script_template.format(
        slurm_scripts_path=slurm_scripts_path,
        file_verification_context=file_verification_context,
        base_path_experiment_repository=base_path_experiment_repository,
        network_folder=network_folder,
        experiment_name=experiment_name,
        epsilon_list=epsilon_list_str,
    )
    with open(temp_slurm_script, "w") as f:
        f.write(slurm_script_content)

```

### Submitting to the Cluster

Once the script is written, `run_slurm_script` makes it executable and invokes `sbatch` to queue the job on the SLURM-managed cluster.

```python

# From examples/scripts/multiple_jobs/main_create_robustness_dist_multiple_jobs.py

def run_slurm_script(temp_slurm_script: Path):
    os.chmod(temp_slurm_script, stat.S_IRWXU)   # make executable

    os.system(f"sbatch {temp_slurm_script}")   # schedule on the cluster

```

## The Verification Worker

Each SLURM node executes [`one_multiple_jobs.py`](https://github.com/ada-research/verona/blob/main/one_multiple_jobs.py), which acts as a stateless worker that reconstructs the verification context and runs the formal analysis.

### Per-Job Execution

The worker receives paths and parameters via command-line arguments, reloads the `ExperimentRepository`, and deserializes the verification context from the YAML file created by the master script.

```python

# From examples/scripts/multiple_jobs/one_multiple_jobs.py

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--file_verification_context", type=Path)
    parser.add_argument("--base_path_experiment_repository")
    parser.add_argument("--network_folder")
    parser.add_argument("--experiment_name")
    parser.add_argument("--epsilon_list", type=float, nargs="+")
    args = parser.parse_args()

    repo = ExperimentRepository(
        base_path=Path(args.base_path_experiment_repository),
        network_folder=Path(args.network_folder),
    )
    repo.load_experiment(experiment_name=args.experiment_name)

    verifier = AutoVerifyModule(verifier=AbCrown(), timeout=360)
    estimator = BinarySearchEpsilonValueEstimator(
        epsilon_value_list=args.epsilon_list.copy(),
        verifier=verifier,
    )
    vc = repo.load_verification_context_from_yaml(
        Path(args.file_verification_context)
    )
    result = estimator.compute_epsilon_value(vc)
    repo.save_result(result)

```

## Aggregating Results and Reporting

After the cluster finishes, the `ExperimentRepository` contains a consolidated `result_df.csv` and optional `per_epsilon_results.csv`. The `ReportCreator` class in [`ada_verona/analysis/report_creator.py`](https://github.com/ada-research/verona/blob/main/ada_verona/analysis/report_creator.py) can then generate visualisations—such as histograms, box-plots, KDE, and ECDF plots—from these aggregated results to analyze the robustness distribution across the network population.

## Summary

- **VERONA** decouples experiment orchestration from verification execution, making it ideal for distributed SLURM clusters.
- The **`ExperimentRepository`** in [`ada_verona/database/experiment_repository.py`](https://github.com/ada-research/verona/blob/main/ada_verona/database/experiment_repository.py) centralizes path management and context serialization via `save_verification_context_to_yaml` and `load_verification_context_from_yaml`.
- **[`main_create_robustness_dist_multiple_jobs.py`](https://github.com/ada-research/verona/blob/main/main_create_robustness_dist_multiple_jobs.py)** generates per-job SLURM scripts using `write_slurm_script` and submits them with `run_slurm_script`.
- Each node executes **[`one_multiple_jobs.py`](https://github.com/ada-research/verona/blob/main/one_multiple_jobs.py)**, which reconstructs the verification context and runs `BinarySearchEpsilonValueEstimator` to compute robustness bounds.
- Results are automatically aggregated into CSV files and can be visualized using **`ReportCreator`**.

## Frequently Asked Questions

### How does VERONA handle file paths across distributed SLURM nodes?

VERONA uses the `ExperimentRepository` class to abstract all file system operations. When the master script creates a job, it serializes the verification context—including absolute paths—to a YAML file using `save_verification_context_to_yaml`. The worker node then calls `load_verification_context_from_yaml` to reconstruct the exact same paths, ensuring consistency across the cluster.

### What verification backend does the SLURM worker use?

The example worker in [`one_multiple_jobs.py`](https://github.com/ada-research/verona/blob/main/one_multiple_jobs.py) instantiates an `AutoVerifyModule` backed by the **AutoCROWN** verifier (`AbCrown` class) with a 360-second timeout. You can swap this for other verifiers supported by VERONA’s modular backend interface, provided they implement the same API used by `BinarySearchEpsilonValueEstimator`.

### How do I customize the SLURM job template?

Modify the `slurm_script_template` string inside [`main_create_robustness_dist_multiple_jobs.py`](https://github.com/ada-research/verona/blob/main/main_create_robustness_dist_multiple_jobs.py). The template uses Python’s `str.format` method to inject paths and the epsilon list. You can adjust `#SBATCH` directives—such as `--partition`, `--exclude`, or GPU requests—to match your cluster’s configuration before the jobs are written and submitted via `sbatch`.

### Where are the results stored after the cluster run completes?

The `ExperimentRepository` automatically persists results to `result_df.csv` (aggregated robustness values) and optionally `per_epsilon_results.csv` (detailed per-epsilon verification outcomes) within the experiment directory. These CSV files can be loaded directly into pandas or processed by `ReportCreator` in [`ada_verona/analysis/report_creator.py`](https://github.com/ada-research/verona/blob/main/ada_verona/analysis/report_creator.py) to generate publication-ready histograms and ECDF plots.