Creating Robustness Distributions from Datasets Using VERONA

VERONA is a modular Python framework that constructs robustness distributions by binary-searching for the largest perturbation ε each data point can tolerate before misclassification, using pluggable verification backends like AutoVerify.

VERONA (Verification-Enabled Robustness Analysis) is an open-source Python framework developed by ada-research for rigorously measuring the ε-robustness of neural network classifiers. Creating robustness distributions from datasets using VERONA allows you to quantify exactly how much adversarial noise each input can withstand before the model changes its prediction, producing statistical safety margins for the entire dataset.

What Is a Robustness Distribution?

A robustness distribution maps every data point in a dataset to its ε-robustness value—the maximum perturbation magnitude (typically L∞ or L₂) that does not cause a misclassification. When aggregated across thousands of points, this forms a statistical distribution revealing the model's safety margins and identifying vulnerable subsets. VERONA automates this process by chaining dataset sampling, property generation, and formal verification into a reproducible pipeline.

Core Components of the VERONA Pipeline

VERONA's architecture decouples data handling, verification, and estimation into interchangeable components. Each component is implemented as a Python class with a specific file path in the ada-research/verona repository.

Dataset Loading with ImageFileDataset

The ImageFileDataset class in ada_verona/database/dataset/image_file_dataset.py loads input images and ground-truth labels from disk. It expects an image folder and a CSV file mapping filenames to class indices, providing the raw data required for robustness analysis.

Sampling Correct Predictions

Not every data point needs verification. The PredictionsBasedSampler in ada_verona/dataset_sampler/predictions_based_sampler.py filters the dataset to include only correctly classified examples (or incorrectly classified ones, if desired). This keeps the verification workload tractable when working with large datasets.

Property Generation for Untargeted Robustness

To verify robustness, VERONA must encode the query as a formal property. The One2AnyPropertyGenerator in ada_verona/verification_module/property_generator/one2any_property_generator.py generates VNN-LIB properties for untargeted robustness—asserting that no perturbation within ε of the input should change the classification.

Verification Backend Integration

The AutoVerifyModule in ada_verona/verification_module/auto_verify_module.py implements the abstract VerificationModule interface defined in ada_verona/verification_module/verification_module.py. It delegates verification to external tools like AbCrown via the AutoVerify library. When using SDP-CROWN, the module injects metadata headers (lines 70-79 of the source) containing the original image, label, and ε value into the VNN-LIB file for traceability.

Binary Search for Epsilon Values

Finding the exact robustness threshold requires searching the ε space. The BinarySearchEpsilonValueEstimator in ada_verona/epsilon_value_estimator/binary_search_epsilon_value_estimator.py implements a binary search over a configurable ε grid. It repeatedly calls the verification module, shrinking the search space based on SAT (counter-example found) or UNSAT (robust) results until it converges on the maximal robust ε.

Experiment Management and Reporting

The ExperimentRepository in ada_verona/database/experiment_repository.py handles persistence. It creates dedicated experiment folders, stores network files, writes VNN-LIB properties, and saves CSV result tables. After processing, save_plots() generates histograms of the robustness distribution. For formal documentation, the optional ReportCreator in ada_verona/analysis/report_creator.py can generate PDF or HTML reports summarizing the statistics.

End-to-End Workflow for Creating Robustness Distributions

The complete pipeline for creating robustness distributions from datasets using VERONA follows these steps:

1. Initialize the dataset – Load images and labels using ImageFileDataset.
2. Sample data points – Use PredictionsBasedSampler to select correctly classified examples.
3. Create verification contexts – For each sampled point, call ExperimentRepository.create_verification_context() to bundle the network, data point, and One2AnyPropertyGenerator.
4. Estimate ε values – Run BinarySearchEpsilonValueEstimator.compute_epsilon_value() to find the robustness threshold via binary search.
5. Persist results – Save EpsilonValueResult objects to the repository.
6. Visualize – Call ExperimentRepository.save_plots() to render the final robustness distribution histograms.

This workflow is fully implemented in the example script examples/scripts/create_robustness_distribution_from_test_dataset.py.

Code Examples

Minimal Programmatic Usage

The following Python code demonstrates the core components needed to create a robustness distribution programmatically:

import logging
import pathlib
from ada_verona.dataset_sampler.predictions_based_sampler import PredictionsBasedSampler
from ada_verona.verification_module.auto_verify_module import AutoVerifyModule
from ada_verona.verification_module.property_generator.one2any_property_generator import One2AnyPropertyGenerator
from ada_verona.epsilon_value_estimator.binary_search_epsilon_value_estimator import BinarySearchEpsilonValueEstimator
from ada_verona.database.experiment_repository import ExperimentRepository
from ada_verona.database.dataset.image_file_dataset import ImageFileDataset
from autoverify.verifier import AbCrown

# Configure logging

logging.basicConfig(level=logging.INFO)
timeout = 600  # seconds per verification

epsilon_grid = [0.001, 0.005, 0.01, 0.02, 0.05]

# Load dataset

dataset = ImageFileDataset(
    image_folder=pathlib.Path("../example_experiment/data/images"),
    label_file=pathlib.Path("../example_experiment/data/image_labels.csv"),
)

# Initialize repository

repo = ExperimentRepository(
    base_path=pathlib.Path("../example_experiment/results"),
    network_folder=pathlib.Path("../example_experiment/data/networks"),
)
repo.initialize_new_experiment("demo")

# Prepare components

prop_gen = One2AnyPropertyGenerator()
verifier = AutoVerifyModule(verifier=AbCrown(), timeout=timeout)
eps_estimator = BinarySearchEpsilonValueEstimator(epsilon_grid, verifier)
sampler = PredictionsBasedSampler(sample_correct_predictions=True)

# Run robustness estimation

for net in repo.get_network_list():
    sampled = sampler.sample(net, dataset)
    
    for dp in sampled:
        ctx = repo.create_verification_context(net, dp, prop_gen)
        eps_result = eps_estimator.compute_epsilon_value(ctx)
        repo.save_result(eps_result)

repo.save_plots()

This example instantiates each component—ImageFileDataset, PredictionsBasedSampler, One2AnyPropertyGenerator, AutoVerifyModule, and BinarySearchEpsilonValueEstimator—and orchestrates them to compute and store robustness distributions. It mirrors the official example script examples/scripts/create_robustness_distribution_from_test_dataset.py.

Command-Line Script Execution

For users who prefer a ready-made solution, VERONA includes a complete example script that requires minimal configuration:

python examples/scripts/create_robustness_distribution_from_test_dataset.py

Executing this script performs the following actions:

1. Loads the images in ../example_experiment/data/images using ImageFileDataset.
2. Builds a One2AnyPropertyGenerator for untargeted robustness queries.
3. Verifies each sampled image with AbCrown across the ε list [0.001, 0.005, 0.01, 0.02, 0.05].
4. Stores a CSV of ε-values per network under ../example_experiment/results.
5. Produces histograms (robustness_distribution.png) via ExperimentRepository.save_plots().

Configuration constants for paths and parameters are located at lines 43-49 of the script source.

Key Implementation Files

Understanding the VERONA source code helps when extending or debugging the robustness distribution pipeline. The following files define the critical interfaces and implementations:

• ada_verona/verification_module/verification_module.py – Abstract base class defining the verify(context, ε) interface that all verifiers must implement.
• ada_verona/verification_module/auto_verify_module.py – Concrete implementation delegating to external verifiers like AbCrown; handles VNN-LIB generation and metadata injection (lines 70-79).
• ada_verona/verification_module/property_generator/one2any_property_generator.py – Generates untargeted robustness properties in VNN-LIB format.
• ada_verona/dataset_sampler/predictions_based_sampler.py – Filters datasets to include only correctly (or incorrectly) classified examples.
• ada_verona/epsilon_value_estimator/binary_search_epsilon_value_estimator.py – Implements binary search to find the exact robustness threshold for each data point.
• ada_verona/database/experiment_repository.py – Manages experiment persistence, including VNN-LIB files, result CSVs, and plot generation.
• ada_verona/analysis/report_creator.py – Optional component for generating PDF/HTML reports from experiment results.
• examples/scripts/create_robustness_distribution_from_test_dataset.py – Complete working example demonstrating the full pipeline.

These files constitute the core of VERONA's robustness-distribution pipeline. By swapping any of the plug-in points (e.g., using a different DatasetSampler or VerificationModule), you can adapt the workflow to new verification back-ends, custom datasets, or alternative robustness notions.

Summary

Creating robustness distributions from datasets using VERONA involves chaining modular components to measure the maximum perturbation each data point can withstand before misclassification. The key takeaways are:

• VERONA is a modular framework that separates data loading, sampling, property generation, verification, and estimation into interchangeable components.
• The binary search estimator (BinarySearchEpsilonValueEstimator) efficiently locates the exact robustness threshold by querying a verification module.
• AutoVerifyModule bridges VERONA to external verifiers like AbCrown, handling VNN-LIB property generation and result parsing.
• The ExperimentRepository provides reproducible experiment management, automatically persisting VNN-LIB files, CSV results, and distribution histograms.
• The complete pipeline is demonstrated in examples/scripts/create_robustness_distribution_from_test_dataset.py.

Frequently Asked Questions

What is the difference between a robustness distribution and individual robustness verification?

Individual robustness verification checks whether a single data point remains correctly classified within a fixed ε-ball. A robustness distribution, by contrast, computes the maximal ε for every point in a dataset, producing a statistical distribution that reveals overall model safety margins and identifies vulnerable subsets. VERONA automates this batch processing via BinarySearchEpsilonValueEstimator.

Can I use a different neural network verifier with VERONA?

Yes. VERONA's VerificationModule abstract base class in ada_verona/verification_module/verification_module.py defines a standard verify(context, ε) interface. You can implement custom verifiers wrapping MILP solvers, SMT engines, or other neural network verification tools. The AutoVerifyModule demonstrates this pattern by delegating to external verifiers like AbCrown.

How does VERONA handle large datasets efficiently?

VERONA uses the PredictionsBasedSampler in ada_verona/dataset_sampler/predictions_based_sampler.py to filter datasets before verification. By default, it selects only correctly classified examples, reducing the verification workload. Additionally, the binary search estimator minimizes verification calls by converging on ε-values logarithmically rather than linearly scanning a grid, making large-scale analysis tractable.

Where are the verification results and plots stored?

The ExperimentRepository in ada_verona/database/experiment_repository.py manages persistence. It creates a dedicated folder per experiment containing VNN-LIB property files, CSV tables of ε-values, and configuration snapshots. After processing, calling save_plots() generates histograms (e.g., robustness_distribution.png) visualizing the robustness distribution across the dataset.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}