# Creating Robustness Distributions from Datasets Using VERONA

> Build robustness distributions from datasets using VERONA a Python framework. Discover the largest perturbation each data point tolerates before misclassification with pluggable verification backends.

- Repository: [ADA research/verona](https://github.com/ada-research/verona)
- Tags: how-to-guide
- Published: 2026-02-23

---

**VERONA is a modular Python framework that constructs robustness distributions by binary-searching for the largest perturbation ε each data point can tolerate before misclassification, using pluggable verification backends like AutoVerify.**

VERONA (Verification-Enabled Robustness Analysis) is an open-source Python framework developed by ada-research for rigorously measuring the ε-robustness of neural network classifiers. Creating robustness distributions from datasets using VERONA allows you to quantify exactly how much adversarial noise each input can withstand before the model changes its prediction, producing statistical safety margins for the entire dataset.

## What Is a Robustness Distribution?

A robustness distribution maps every data point in a dataset to its **ε-robustness value**—the maximum perturbation magnitude (typically L∞ or L₂) that does not cause a misclassification. When aggregated across thousands of points, this forms a statistical distribution revealing the model's safety margins and identifying vulnerable subsets. VERONA automates this process by chaining dataset sampling, property generation, and formal verification into a reproducible pipeline.

## Core Components of the VERONA Pipeline

VERONA's architecture decouples data handling, verification, and estimation into interchangeable components. Each component is implemented as a Python class with a specific file path in the ada-research/verona repository.

### Dataset Loading with ImageFileDataset

The `ImageFileDataset` class in [`ada_verona/database/dataset/image_file_dataset.py`](https://github.com/ada-research/verona/blob/main/ada_verona/database/dataset/image_file_dataset.py) loads input images and ground-truth labels from disk. It expects an image folder and a CSV file mapping filenames to class indices, providing the raw data required for robustness analysis.

### Sampling Correct Predictions

Not every data point needs verification. The `PredictionsBasedSampler` in [`ada_verona/dataset_sampler/predictions_based_sampler.py`](https://github.com/ada-research/verona/blob/main/ada_verona/dataset_sampler/predictions_based_sampler.py) filters the dataset to include only correctly classified examples (or incorrectly classified ones, if desired). This keeps the verification workload tractable when working with large datasets.

### Property Generation for Untargeted Robustness

To verify robustness, VERONA must encode the query as a formal property. The `One2AnyPropertyGenerator` in [`ada_verona/verification_module/property_generator/one2any_property_generator.py`](https://github.com/ada-research/verona/blob/main/ada_verona/verification_module/property_generator/one2any_property_generator.py) generates VNN-LIB properties for untargeted robustness—asserting that no perturbation within ε of the input should change the classification.

### Verification Backend Integration

The `AutoVerifyModule` in [`ada_verona/verification_module/auto_verify_module.py`](https://github.com/ada-research/verona/blob/main/ada_verona/verification_module/auto_verify_module.py) implements the abstract `VerificationModule` interface defined in [`ada_verona/verification_module/verification_module.py`](https://github.com/ada-research/verona/blob/main/ada_verona/verification_module/verification_module.py). It delegates verification to external tools like **AbCrown** via the AutoVerify library. When using SDP-CROWN, the module injects metadata headers (lines 70-79 of the source) containing the original image, label, and ε value into the VNN-LIB file for traceability.

### Binary Search for Epsilon Values

Finding the exact robustness threshold requires searching the ε space. The `BinarySearchEpsilonValueEstimator` in [`ada_verona/epsilon_value_estimator/binary_search_epsilon_value_estimator.py`](https://github.com/ada-research/verona/blob/main/ada_verona/epsilon_value_estimator/binary_search_epsilon_value_estimator.py) implements a binary search over a configurable ε grid. It repeatedly calls the verification module, shrinking the search space based on SAT (counter-example found) or UNSAT (robust) results until it converges on the maximal robust ε.

### Experiment Management and Reporting

The `ExperimentRepository` in [`ada_verona/database/experiment_repository.py`](https://github.com/ada-research/verona/blob/main/ada_verona/database/experiment_repository.py) handles persistence. It creates dedicated experiment folders, stores network files, writes VNN-LIB properties, and saves CSV result tables. After processing, `save_plots()` generates histograms of the robustness distribution. For formal documentation, the optional `ReportCreator` in [`ada_verona/analysis/report_creator.py`](https://github.com/ada-research/verona/blob/main/ada_verona/analysis/report_creator.py) can generate PDF or HTML reports summarizing the statistics.

## End-to-End Workflow for Creating Robustness Distributions

The complete pipeline for creating robustness distributions from datasets using VERONA follows these steps:

1. **Initialize the dataset** – Load images and labels using `ImageFileDataset`.
2. **Sample data points** – Use `PredictionsBasedSampler` to select correctly classified examples.
3. **Create verification contexts** – For each sampled point, call `ExperimentRepository.create_verification_context()` to bundle the network, data point, and `One2AnyPropertyGenerator`.
4. **Estimate ε values** – Run `BinarySearchEpsilonValueEstimator.compute_epsilon_value()` to find the robustness threshold via binary search.
5. **Persist results** – Save `EpsilonValueResult` objects to the repository.
6. **Visualize** – Call `ExperimentRepository.save_plots()` to render the final robustness distribution histograms.

This workflow is fully implemented in the example script [`examples/scripts/create_robustness_distribution_from_test_dataset.py`](https://github.com/ada-research/verona/blob/main/examples/scripts/create_robustness_distribution_from_test_dataset.py).

## Code Examples

### Minimal Programmatic Usage

The following Python code demonstrates the core components needed to create a robustness distribution programmatically:

```python
import logging
import pathlib
from ada_verona.dataset_sampler.predictions_based_sampler import PredictionsBasedSampler
from ada_verona.verification_module.auto_verify_module import AutoVerifyModule
from ada_verona.verification_module.property_generator.one2any_property_generator import One2AnyPropertyGenerator
from ada_verona.epsilon_value_estimator.binary_search_epsilon_value_estimator import BinarySearchEpsilonValueEstimator
from ada_verona.database.experiment_repository import ExperimentRepository
from ada_verona.database.dataset.image_file_dataset import ImageFileDataset
from autoverify.verifier import AbCrown

# Configure logging

logging.basicConfig(level=logging.INFO)
timeout = 600  # seconds per verification

epsilon_grid = [0.001, 0.005, 0.01, 0.02, 0.05]

# Load dataset

dataset = ImageFileDataset(
    image_folder=pathlib.Path("../example_experiment/data/images"),
    label_file=pathlib.Path("../example_experiment/data/image_labels.csv"),
)

# Initialize repository

repo = ExperimentRepository(
    base_path=pathlib.Path("../example_experiment/results"),
    network_folder=pathlib.Path("../example_experiment/data/networks"),
)
repo.initialize_new_experiment("demo")

# Prepare components

prop_gen = One2AnyPropertyGenerator()
verifier = AutoVerifyModule(verifier=AbCrown(), timeout=timeout)
eps_estimator = BinarySearchEpsilonValueEstimator(epsilon_grid, verifier)
sampler = PredictionsBasedSampler(sample_correct_predictions=True)

# Run robustness estimation

for net in repo.get_network_list():
    sampled = sampler.sample(net, dataset)
    
    for dp in sampled:
        ctx = repo.create_verification_context(net, dp, prop_gen)
        eps_result = eps_estimator.compute_epsilon_value(ctx)
        repo.save_result(eps_result)

repo.save_plots()

```

This example instantiates each component—`ImageFileDataset`, `PredictionsBasedSampler`, `One2AnyPropertyGenerator`, `AutoVerifyModule`, and `BinarySearchEpsilonValueEstimator`—and orchestrates them to compute and store robustness distributions. It mirrors the official example script [`examples/scripts/create_robustness_distribution_from_test_dataset.py`](https://github.com/ada-research/verona/blob/main/examples/scripts/create_robustness_distribution_from_test_dataset.py).

### Command-Line Script Execution

For users who prefer a ready-made solution, VERONA includes a complete example script that requires minimal configuration:

```bash
python examples/scripts/create_robustness_distribution_from_test_dataset.py

```

Executing this script performs the following actions:

1. Loads the images in `../example_experiment/data/images` using `ImageFileDataset`.
2. Builds a `One2AnyPropertyGenerator` for untargeted robustness queries.
3. Verifies each sampled image with **AbCrown** across the ε list `[0.001, 0.005, 0.01, 0.02, 0.05]`.
4. Stores a CSV of ε-values per network under `../example_experiment/results`.
5. Produces histograms (`robustness_distribution.png`) via `ExperimentRepository.save_plots()`.

Configuration constants for paths and parameters are located at lines 43-49 of the script source.

## Key Implementation Files

Understanding the VERONA source code helps when extending or debugging the robustness distribution pipeline. The following files define the critical interfaces and implementations:

- **[`ada_verona/verification_module/verification_module.py`](https://github.com/ada-research/verona/blob/main/ada_verona/verification_module/verification_module.py)** – Abstract base class defining the `verify(context, ε)` interface that all verifiers must implement.
- **[`ada_verona/verification_module/auto_verify_module.py`](https://github.com/ada-research/verona/blob/main/ada_verona/verification_module/auto_verify_module.py)** – Concrete implementation delegating to external verifiers like AbCrown; handles VNN-LIB generation and metadata injection (lines 70-79).
- **[`ada_verona/verification_module/property_generator/one2any_property_generator.py`](https://github.com/ada-research/verona/blob/main/ada_verona/verification_module/property_generator/one2any_property_generator.py)** – Generates untargeted robustness properties in VNN-LIB format.
- **[`ada_verona/dataset_sampler/predictions_based_sampler.py`](https://github.com/ada-research/verona/blob/main/ada_verona/dataset_sampler/predictions_based_sampler.py)** – Filters datasets to include only correctly (or incorrectly) classified examples.
- **[`ada_verona/epsilon_value_estimator/binary_search_epsilon_value_estimator.py`](https://github.com/ada-research/verona/blob/main/ada_verona/epsilon_value_estimator/binary_search_epsilon_value_estimator.py)** – Implements binary search to find the exact robustness threshold for each data point.
- **[`ada_verona/database/experiment_repository.py`](https://github.com/ada-research/verona/blob/main/ada_verona/database/experiment_repository.py)** – Manages experiment persistence, including VNN-LIB files, result CSVs, and plot generation.
- **[`ada_verona/analysis/report_creator.py`](https://github.com/ada-research/verona/blob/main/ada_verona/analysis/report_creator.py)** – Optional component for generating PDF/HTML reports from experiment results.
- **[`examples/scripts/create_robustness_distribution_from_test_dataset.py`](https://github.com/ada-research/verona/blob/main/examples/scripts/create_robustness_distribution_from_test_dataset.py)** – Complete working example demonstrating the full pipeline.

These files constitute the core of VERONA's robustness-distribution pipeline. By swapping any of the plug-in points (e.g., using a different `DatasetSampler` or `VerificationModule`), you can adapt the workflow to new verification back-ends, custom datasets, or alternative robustness notions.

## Summary

Creating robustness distributions from datasets using VERONA involves chaining modular components to measure the maximum perturbation each data point can withstand before misclassification. The key takeaways are:

- **VERONA** is a modular framework that separates data loading, sampling, property generation, verification, and estimation into interchangeable components.
- The **binary search estimator** (`BinarySearchEpsilonValueEstimator`) efficiently locates the exact robustness threshold by querying a verification module.
- **AutoVerifyModule** bridges VERONA to external verifiers like AbCrown, handling VNN-LIB property generation and result parsing.
- The **ExperimentRepository** provides reproducible experiment management, automatically persisting VNN-LIB files, CSV results, and distribution histograms.
- The complete pipeline is demonstrated in [`examples/scripts/create_robustness_distribution_from_test_dataset.py`](https://github.com/ada-research/verona/blob/main/examples/scripts/create_robustness_distribution_from_test_dataset.py).

## Frequently Asked Questions

### What is the difference between a robustness distribution and individual robustness verification?

Individual robustness verification checks whether a single data point remains correctly classified within a fixed ε-ball. A robustness distribution, by contrast, computes the *maximal* ε for every point in a dataset, producing a statistical distribution that reveals overall model safety margins and identifies vulnerable subsets. VERONA automates this batch processing via `BinarySearchEpsilonValueEstimator`.

### Can I use a different neural network verifier with VERONA?

Yes. VERONA's `VerificationModule` abstract base class in [`ada_verona/verification_module/verification_module.py`](https://github.com/ada-research/verona/blob/main/ada_verona/verification_module/verification_module.py) defines a standard `verify(context, ε)` interface. You can implement custom verifiers wrapping MILP solvers, SMT engines, or other neural network verification tools. The `AutoVerifyModule` demonstrates this pattern by delegating to external verifiers like AbCrown.

### How does VERONA handle large datasets efficiently?

VERONA uses the `PredictionsBasedSampler` in [`ada_verona/dataset_sampler/predictions_based_sampler.py`](https://github.com/ada-research/verona/blob/main/ada_verona/dataset_sampler/predictions_based_sampler.py) to filter datasets before verification. By default, it selects only correctly classified examples, reducing the verification workload. Additionally, the binary search estimator minimizes verification calls by converging on ε-values logarithmically rather than linearly scanning a grid, making large-scale analysis tractable.

### Where are the verification results and plots stored?

The `ExperimentRepository` in [`ada_verona/database/experiment_repository.py`](https://github.com/ada-research/verona/blob/main/ada_verona/database/experiment_repository.py) manages persistence. It creates a dedicated folder per experiment containing VNN-LIB property files, CSV tables of ε-values, and configuration snapshots. After processing, calling `save_plots()` generates histograms (e.g., `robustness_distribution.png`) visualizing the robustness distribution across the dataset.