How to Export Verification Results to ExperimentRepository in VERONA

Exporting verification results to ExperimentRepository in VERONA involves calling repo.save_result() after running AutoVerifyModule.verify(), which persists outcomes to CSV files, generates plots via ReportCreator, and stores per-ε data for reproducible robustness experiments.

VERONA is an open-source framework for neural network robustness verification developed by ADA Research. When executing verification experiments, the ExperimentRepository class serves as the central database that orchestrates directory creation, persists results, and generates visualizations. This guide explains how to export verification results to ExperimentRepository in VERONA using the AutoVerifyModule and the complete data persistence pipeline.

Understanding the VERONA Verification Pipeline

The verification pipeline in VERONA consists of loosely-coupled components defined in ada_verona/verification_module/verification_module.py. The abstract VerificationModule interface defines the verify contract, while concrete implementations like AutoVerifyModule handle external verifier integration.

The pipeline flow involves four primary components:

• VerificationContext (ada_verona/database/verification_context.py) – Bundles the network, data point, temporary folder, and property generator.
• AutoVerifyModule (ada_verona/verification_module/auto_verify_module.py) – Builds VNNLIB properties, injects VERONA metadata, and calls external verifiers.
• CompleteVerificationData (ada_verona/database/verification_result.py) – Dataclass returned upon successful verification containing outcomes and counter-examples.
• ExperimentRepository (ada_verona/database/experiment_repository.py) – Persists results to CSV, generates plots, and saves contexts as YAML.

When AutoVerifyModule.verify() completes, it returns either a CompleteVerificationData object wrapped in a Result.Ok or an error string. The repository then persists this data across multiple targets including aggregated CSV files, per-epsilon CSVs, visualization plots, and YAML context files.

Step-by-Step Guide to Exporting Results

The following workflow demonstrates how to initialize an experiment, run verification for a specific epsilon perturbation, and export the results to the repository:

from pathlib import Path
from ada_verona.database.experiment_repository import ExperimentRepository
from ada_verona.verification_module.auto_verify_module import AutoVerifyModule
from ada_verona.database.dataset.data_point import DataPoint
from ada_verona.verification_module.property_generator.one2any_property_generator import One2AnyPropertyGenerator
from ada_verona.database.machine_learning_model.onnx_network import ONNXNetwork
from autoverify.verifier.verifier import Verifier

# 1. Initialize the experiment repository

repo = ExperimentRepository(
    base_path=Path("experiments"),
    network_folder=Path("networks")
)
repo.initialize_new_experiment("mnist_demo")

# 2. Load network and data point

network = ONNXNetwork(Path("examples/example_experiment/data/networks/mnist-net_256x2.onnx"))
image = Path("examples/example_experiment/data/images/mnist_train_0.pt")
data_point = DataPoint(label=5, data=load(image), id="0")

# 3. Create verification context

prop_gen = One2AnyPropertyGenerator()
ctx = repo.create_verification_context(network, data_point, prop_gen)

# 4. Initialize verifier

verifier = Verifier(name="sdpcrown")
auto_mod = AutoVerifyModule(verifier=verifier, timeout=300)

# 5. Run verification

epsilon = 0.05
result = auto_mod.verify(ctx, epsilon)

# 6. Export to repository

if isinstance(result, dict) or hasattr(result, "result"):
    repo.save_result(result)
else:
    print("Verification failed:", result)

# 7. Generate reports

repo.save_plots()
repo.save_per_epsilon_result_df()

Persistence Targets in ExperimentRepository

When exporting verification results to ExperimentRepository in VERONA, the system creates four distinct persistence targets that together enable comprehensive analysis and reproducibility.

Results CSV

The Results CSV (results/result_df.csv) is created by ExperimentRepository.save_result(s). It contains one row per ε-run with columns for the network path, epsilon value, verification outcome, and timestamps.

Per-ε CSV

The Per-ε CSV (tmp/<network>/image_<id>/epsilons_df.csv) stores the full list of ε values explored for a single image. These files are aggregated via ExperimentRepository.save_per_epsilon_result_df() for fine-grained analysis of robustness boundaries.

Visualization Plots

Generated by ExperimentRepository.save_plots() using ReportCreator from ada_verona/analysis/report_creator.py, the following plots are created in the results/ directory:

• hist_figure.png – Histogram of verification outcomes
• boxplot.png – Distribution of epsilon values by outcome
• kde_plot.png – Kernel density estimation of results
• ecdf_plot.png – Empirical cumulative distribution function

Context YAML

The Context YAML (tmp/<network>/image_<id>/context.yaml) is created by ExperimentRepository.save_verification_context_to_yaml(). This file enables full reproducibility by storing the complete verification context, allowing specific runs to be re-executed or debugged later.

Metadata Injection for Reproducibility

When using the SDP-CROWN verifier, AutoVerifyModule.verify() injects VERONA-specific metadata directly into the VNNLIB property file. This occurs in ada_verona/verification_module/auto_verify_module.py (lines 67-80):

if (self.verifier.name == "sdpcrown") and ("; verona_metadata_version:" not in vnnlib_property.content):
    image_flat = verification_context.data_point.data.detach().cpu().numpy().reshape(-1)
    image_csv = ",".join(f"{v:.8f}" for v in image_flat)
    header = (
        "; verona_metadata_version: 1\n"
        f"; verona_epsilon: {float(epsilon):.8f}\n"
        f"; verona_image_class: {int(verification_context.data_point.label)}\n"
        f"; verona_image: {image_csv}\n"
    )
    vnnlib_property.content = header + vnnlib_property.content

This header enables downstream tools to parse verification parameters including the exact epsilon value, image class label, and flattened pixel data, facilitating attack reproduction and result validation.

Summary

• ExperimentRepository in VERONA acts as the central database for robustness experiments, managing directory structures and result persistence across multiple formats.
• Call repo.save_result() to append verification outcomes to results/result_df.csv after successful AutoVerifyModule.verify() execution.
• Use repo.save_plots() to generate histograms, box plots, KDE, and ECDF visualizations via the ReportCreator class.
• Per-ε data is stored in temporary CSV files and aggregated using repo.save_per_epsilon_result_df() for detailed boundary analysis.
• Full reproducibility is supported through YAML context serialization via save_verification_context_to_yaml().
• VERONA metadata is embedded directly into VNNLIB files when using SDP-CROWN, storing epsilon values, image classes, and flattened image data for downstream parsing.

Frequently Asked Questions

What file format does VERONA use for storing verification results?

VERONA stores aggregated verification results in CSV format at results/result_df.csv. Each row represents a single verification run with columns for the network path, epsilon value, verification outcome, and timestamps. Per-epsilon explorations are stored in separate CSV files within the temporary experiment folder structure under tmp/<network>/image_<id>/epsilons_df.csv.

How does ExperimentRepository handle failed verifications?

When AutoVerifyModule.verify() returns an error string rather than CompleteVerificationData, the repository's save_result() method should not be called with the error object. Instead, implement conditional logic to check isinstance(result, dict) or hasattr(result, "result") before invoking repo.save_result() to ensure only valid data is persisted to the CSV files.

Can I regenerate plots without re-running verifications?

Yes. Since ExperimentRepository.save_plots() reads from the accumulated results/result_df.csv file and utilizes ReportCreator from ada_verona/analysis/report_creator.py, you can regenerate histograms, box plots, KDE plots, and ECDF plots at any time after results have been saved, without re-executing the verification modules or reloading neural networks.

What is the purpose of the VERONA metadata header in VNNLIB files?

The metadata header injected by AutoVerifyModule stores the epsilon value, image class label, and flattened image pixel values as semicolon-prefixed comments in the VNNLIB property file. This allows downstream analysis tools and attack reproduction scripts to parse the exact verification parameters and input data used during the robustness check, ensuring experimental reproducibility.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}