How to Implement Custom Epsilon Value Estimators in VERONA

To implement a custom epsilon value estimator in VERONA, subclass the abstract EpsilonValueEstimator base class and implement the compute_epsilon_value method to orchestrate your search strategy over candidate perturbation bounds.

VERONA is an open-source robustness verification framework that estimates the critical epsilon value—the largest adversarial perturbation that keeps a neural network verification problem UNSAT—for a given safety property. While the repository ships with binary search and iterative strategies, the modular architecture in ada-research/verona allows you to implement custom epsilon value estimators by extending the base class and leveraging the VerificationContext API.

Understanding the EpsilonValueEstimator Architecture

The estimation workflow centers on the EpsilonValueEstimator abstract base class defined in ada_verona/epsilon_value_estimator/epsilon_value_estimator.py. This class standardizes how VERONA searches through candidate epsilon values to find the robustness boundary.

Core Components

Every custom estimator interacts with the following key classes:

• EpsilonValueEstimator (ada_verona/epsilon_value_estimator/epsilon_value_estimator.py): The abstract base class that stores epsilon_value_list and verifier in its constructor and declares the abstract compute_epsilon_value method.
• VerificationContext (ada_verona/database/verification_context.py): Bundles the network, data point, property generator, and result log. Provides save_result() to persist intermediate trials.
• EpsilonStatus (ada_verona/database/epsilon_status.py): Records individual epsilon trials including the value, result, runtime, and verifier name.
• EpsilonValueResult (ada_verona/database/epsilon_value_result.py): The final container returned by your estimator, containing the critical epsilon, smallest SAT epsilon, total runtime, and verification context reference.
• VerificationResult (ada_verona/database/verification_result.py): Enum providing standardized outcome codes (SAT, UNSAT, UNKNOWN).

Built-in Reference Implementations

VERONA provides two concrete implementations that demonstrate the expected interface:

1. BinarySearchEpsilonValueEstimator (ada_verona/epsilon_value_estimator/binary_search_epsilon_value_estimator.py): Performs classic binary search over the supplied epsilon list, returning the highest UNSAT and smallest SAT values.
2. IterativeEpsilonValueEstimator (ada_verona/epsilon_value_estimator/iterative_epsilon_value_estimator.py): Linearly scans the sorted epsilon list and records extreme UNSAT/SAT values.

These classes are re-exported in ada_verona/__init__.py for convenient import.

Building a Custom Epsilon Estimator

To create a bespoke estimation strategy—such as exponential search, adaptive sampling, or learned prediction—you must implement the compute_epsilon_value method to iterate over self.epsilon_value_list and call self.verifier.verify().

Step 1: Subclass and Implement the Interface

Create a new class that inherits from EpsilonValueEstimator and implements compute_epsilon_value. The method must accept a VerificationContext object and return an EpsilonValueResult.

# my_custom_estimator.py

from ada_verona.epsilon_value_estimator.epsilon_value_estimator import EpsilonValueEstimator
from ada_verona.database.epsilon_value_result import EpsilonValueResult
from ada_verona.database.epsilon_status import EpsilonStatus
from ada_verona.database.verification_result import VerificationResult
import logging
import time

logger = logging.getLogger(__name__)

class ExponentialBinaryEpsilonEstimator(EpsilonValueEstimator):
    """
    First exponentially increase ε until a SAT is observed,
    then perform a binary search between the last UNSAT and the first SAT.
    """

    def compute_epsilon_value(self, verification_context) -> EpsilonValueResult:
        # Exponential phase

        factor = 2.0
        current = min(self.epsilon_value_list)
        last_unsat = None
        first_sat = None

        while True:
            outcome = self.verifier.verify(verification_context, current)
            status = EpsilonStatus(current, None, None, self.verifier.name)
            status.set_values(outcome)
            verification_context.save_result(status)

            logger.debug("ε=%s → %s", current, outcome)
            if outcome == VerificationResult.UNSAT:
                last_unsat = current
                current *= factor
            else:   # SAT or UNKNOWN → treat as SAT for interval creation

                first_sat = current
                break

        # Binary search inside the interval [last_unsat, first_sat]

        epsilon_candidates = sorted(
            [last_unsat, first_sat] if last_unsat is not None else [first_sat]
        )
        
        # Re-use the existing binary-search implementation for robustness

        from ada_verona.epsilon_value_estimator.binary_search_epsilon_value_estimator import (
            BinarySearchEpsilonValueEstimator,
        )

        full_list = sorted(set(self.epsilon_value_list + epsilon_candidates))
        binary_est = BinarySearchEpsilonValueEstimator(
            epsilon_value_list=full_list, verifier=self.verifier
        )
        return binary_est.compute_epsilon_value(verification_context)

Step 2: Handle Verification Results and Context

Inside compute_epsilon_value, you must:

• Call self.verifier.verify(context, epsilon) for each candidate value
• Create an EpsilonStatus instance for each trial
• Persist intermediate results using verification_context.save_result(status)
• Aggregate outcomes to determine the critical epsilon

The EpsilonStatus.set_values(outcome) method automatically records the verification result and timing information.

Step 3: Return the Final Result

Your implementation must return an EpsilonValueResult object containing the highest UNSAT epsilon, the smallest SAT epsilon (if found), total runtime, and a reference to the VerificationContext. You can instantiate this directly or delegate to existing estimators as shown in the exponential search example above.

Integrating Custom Estimators into VERONA Workflows

Once implemented, your custom estimator integrates seamlessly with VERONA's experiment infrastructure. Instantiate it exactly like built-in estimators and pass it to your verification loop.

# example_usage.py

import logging
from pathlib import Path

from ada_verona.verification_module.auto_verify_module import AutoVerifyModule
from ada_verona.database.experiment_repository import ExperimentRepository
from ada_verona.dataset_sampler.predictions_based_sampler import PredictionsBasedSampler
from ada_verona.verification_module.property_generator.one2any_property_generator import (
    One2AnyPropertyGenerator,
)
from ada_verona.database.dataset.image_file_dataset import ImageFileDataset
from ada_verona.epsilon_value_estimator.epsilon_value_estimator import EpsilonValueEstimator

# Import the custom estimator

from my_custom_estimator import ExponentialBinaryEpsilonEstimator

logging.basicConfig(level=logging.INFO)

# Set up experiment infrastructure

repo = ExperimentRepository(base_path=Path("./results"), network_folder=Path("./networks"))
repo.initialize_new_experiment("my_exp")
dataset = ImageFileDataset(image_folder=Path("./images"),
                           label_file=Path("./labels.csv"))
property_gen = One2AnyPropertyGenerator()

# Create verifier (AutoVerify with AbCrown)

from autoverify.verifier import AbCrown
verifier = AutoVerifyModule(verifier=AbCrown(), timeout=300)

# Initialize the custom estimator

epsilon_candidates = [0.001, 0.005, 0.01, 0.02, 0.05, 0.1]
estimator: EpsilonValueEstimator = ExponentialBinaryEpsilonEstimator(
    epsilon_value_list=epsilon_candidates,
    verifier=verifier,
)

# Run the estimation for each network and data point

sampler = PredictionsBasedSampler(sample_correct_predictions=True)

for net in repo.get_network_list():
    for dp in sampler.sample(net, dataset):
        ctx = repo.create_verification_context(net, dp, property_gen)
        result = estimator.compute_epsilon_value(ctx)
        repo.save_result(result)

repo.save_plots()

The custom estimator receives the same epsilon_value_list and verifier objects as built-in implementations, allowing seamless swapping between strategies without modifying the surrounding experiment code.

Summary

To implement custom epsilon value estimators in VERONA:

• Subclass EpsilonValueEstimator from ada_verona/epsilon_value_estimator/epsilon_value_estimator.py
• Implement compute_epsilon_value to define your search logic using the injected verifier and VerificationContext
• Persist intermediate results via context.save_result(EpsilonStatus(...)) for experiment tracking
• Return an EpsilonValueResult containing the critical epsilon and boundary values
• Compose existing estimators like BinarySearchEpsilonValueEstimator to avoid reimplementing low-level logic

Frequently Asked Questions

What methods must I implement when subclassing EpsilonValueEstimator?

You must implement the abstract method compute_epsilon_value(self, verification_context) which accepts a VerificationContext object and returns an EpsilonValueResult. The base class constructor already handles storing the epsilon_value_list and verifier parameters as instance attributes (self.epsilon_value_list and self.verifier).

Can I compose existing estimators within my custom implementation?

Yes. As demonstrated in the ExponentialBinaryEpsilonEstimator example, you can import and instantiate built-in estimators like BinarySearchEpsilonValueEstimator within your compute_epsilon_value method. This pattern allows you to implement coarse-to-fine strategies or fallback logic while reusing verified binary search or iterative scanning implementations from ada_verona/epsilon_value_estimator/.

How does the VerificationContext track intermediate results?

The VerificationContext class maintains a result log for each epsilon trial. Inside your estimator, create an EpsilonStatus object for each verification call, populate it with status.set_values(outcome), and persist it using verification_context.save_result(status). This automatically records the epsilon value, verification outcome, runtime, and verifier name to the experiment database.

Where should I save my custom estimator class files?

You can save custom estimator modules anywhere in your project directory, provided they can import from ada_verona. For production workflows, place them in a dedicated package (e.g., my_project/custom_estimators/) and import them into your experiment scripts. Ensure your custom class inherits from EpsilonValueEstimator and follows the method signature conventions defined in ada_verona/epsilon_value_estimator/epsilon_value_estimator.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}