Debugging Verification Timeouts and Performance Issues in VERONA

Most verification timeouts in VERONA originate from three sources: the timeout value passed to external verifiers, oversized VNNLIB property headers containing full image CSV metadata, and blocking I/O during property serialization.

VERONA, developed by ada-research, is an open-source framework designed for scalable neural network verification. When running robustness analysis on high-dimensional inputs or large datasets, developers frequently encounter premature timeouts, excessive memory consumption, and verifier hangs. Understanding how the AutoVerifyModule propagates timeout limits and manages property generation is essential for diagnosing and resolving these performance bottlenecks.

How VERONA Applies Verification Timeouts

VERONA’s verification workflow relies on three core abstractions that handle timeout management:

• Verification Context (ada_verona/database/verification_context.py) – Bundles the neural network, data point, and property generator.
• Verification Module (ada_verona/verification_module/verification_module.py) – Defines the abstract verify API that concrete verifiers must implement.
• AutoVerifyModule (ada_verona/verification_module/auto_verify_module.py) – Drives external verifiers like SDP-CROWN or Marabou and manages timeout enforcement, configuration files, and result parsing.

Timeout Propagation Flow

The timeout mechanism operates across three distinct phases:

1. Construction – The caller supplies a timeout value in seconds when instantiating AutoVerifyModule. This value is stored in self.timeout at line 46 of auto_verify_module.py.

2. Invocation – When the verify method is called, the module forwards the timeout to the underlying verifier’s verify_property method (lines 84–95).

3. Verifier Execution – The external verifier (implemented in the autoverify package) receives the timeout and aborts the process if the wall-clock limit is reached.

Because the timeout is a simple float passed through these layers, most performance-related bugs stem from configuration errors rather than framework logic.

Common Performance Bottlenecks

When debugging verification timeouts, identify your specific symptom to locate the root cause:

Symptom: Verifier hangs for minutes despite a small timeout value

• Likely Cause: The external verifier ignores the timeout parameter. Some verifiers only support wall-clock arguments or require specific configuration flags.
• Where to Look: Inspect the concrete verifier class in autoverify.verifier.verifier.Verifier to confirm timeout handling.

Symptom: Large memory usage before timeout occurs

• Likely Cause: The property generator embeds the full image as CSV data in the VNNLIB header (lines 70–78 of verification_context.py). For large images, this header can grow to several megabytes.
• Where to Look: Examine the generated .vnnlib file size and content.

Symptom: Frequent "Error during verification" log messages

• Likely Cause: AutoVerifyModule catches Err results and returns the error string (lines 108–110). The underlying result.unwrap_err() often contains the original timeout exception masked as a generic error.
• Where to Look: Enable debug logging to capture the raw exception from the verifier.

Symptom: Slow performance during repeated verification calls

• Likely Cause: AutoVerifyModule recreates the VNNLIB file on every verify call, causing significant I/O overhead when the same (image, epsilon) pair appears frequently.
• Where to Look: Profile file system operations during batch verification runs.

Debugging Verification Timeouts Step-by-Step

Follow this systematic approach to isolate timeout and performance issues:

Verify Timeout Propagation

Enable debug logging to confirm the timeout value reaches the verifier:

import logging
logging.getLogger('ada_verona.verification_module.auto_verify_module').setLevel(logging.DEBUG)

The debug output will display the exact timeout= argument passed at lines 84–95 of auto_verify_module.py.

Inspect the Generated VNNLIB File

Check for oversized metadata headers that slow down parsing:

ctx.save_vnnlib_property(vnnlib)  # inside verify()

with open(vnnlib.path) as f:
    print(f.read()[:500])   # header + first 10 constraints

If the header contains a massive CSV list (lines 71–78), reduce image resolution or disable metadata embedding via feature flags.

Measure Actual Verifier Runtime

Compare elapsed time against the configured timeout to detect non-compliance:

import time
start = time.time()
result = self.verifier.verify_property(..., timeout=self.timeout)
print("Verifier elapsed:", time.time() - start)

A large discrepancy between elapsed time and self.timeout indicates the verifier is not respecting the limit.

Tune Timeout Values by Attack Type

Adjust timeouts based on the verification algorithm:

• FGSM attacks (fgsm_attack.py): Typically complete in under 5 seconds; set timeout=30.
• PGD attacks (pgd_attack.py): Iterative nature requires 300–600 seconds for convergence.

Profile Property Generation

If CPU usage spikes before verification begins, profile the property generator:

from ada_verona.verification_module.property_generator.property_generator import PropertyGenerator
import cProfile, pstats

pr = cProfile.Profile()
pr.enable()
vnn = pg.create_vnnlib_property(image, label, epsilon)
pr.disable()
pstats.Stats(pr).sort_stats('cumtime').print_stats(10)

Architectural Levers for Better Performance

Optimize throughput using these framework features:

Batch Verification

AutoVerifyModule processes single data points. For large datasets, use the multiple-jobs scripts in examples/scripts/multiple_jobs/ to launch parallel AutoVerifyModule instances.

Verifier Configuration

Pass a config file (see examples/scripts/create_robustness_dist_*.py) to adjust verifier-specific parameters like max_iterations or use_gpu for GPU-accelerated solving.

Result Caching

Store CompleteVerificationData objects in the database (ada_verona/database/verification_result.py) to avoid re-running verification on identical epsilons.

Parallel Property Generation

The property generator is pure Python. Wrap calls in concurrent.futures.ThreadPoolExecutor if property creation dominates runtime during batch processing.

Practical Code Examples

Running Verification with a Custom Timeout

from pathlib import Path
from ada_verona.verification_module.auto_verify_module import AutoVerifyModule
from ada_verona.database.verification_context import VerificationContext
from autoverify.verifier.verifier import VerifierFactory

# Load a verifier (e.g. SDP-CROWN)

verifier = VerifierFactory.create("sdpcrown")

# Increase timeout to 10 minutes for complex properties

auto_mod = AutoVerifyModule(verifier=verifier, timeout=600)

# Build verification context

ctx = VerificationContext.load(
    network_path=Path("models/resnet18.onnx"),
    dataset_path=Path("data/mnist_test.pt"),
    property_generator="one2one"
)

# Verify at epsilon = 0.02

result = auto_mod.verify(ctx, epsilon=0.02)
print(result)   # → CompleteVerificationData or error string

Detecting Timeout-Related Errors

from ada_verona.verification_module.auto_verify_module import AutoVerifyModule
import logging

logging.basicConfig(level=logging.INFO)

result = auto_mod.verify(ctx, epsilon=0.05)
if isinstance(result, str) and "timeout" in result.lower():
    print("Verification hit the timeout limit – consider raising it.")

Profiling Property Generation

import cProfile, pstats
from ada_verona.verification_module.property_generator.one2one_property_generator import One2OnePropertyGenerator
from ada_verona.database.verification_context import VerificationContext

pg = One2OnePropertyGenerator()
ctx = VerificationContext.load(...)
image = ctx.data_point.data.reshape(-1).detach().numpy()

pr = cProfile.Profile()
pr.enable()
_ = pg.create_vnnlib_property(image, ctx.data_point.label, epsilon=0.01)
pr.disable()
pstats.Stats(pr).sort_stats('cumtime').print_stats(5)

Summary

• Timeout propagation flows from AutoVerifyModule construction (line 46) through to the external verifier’s verify_property method (lines 84–95), but external verifiers may not always respect the wall-clock limit.
• Memory bottlenecks typically occur when verification_context.py embeds full image data as CSV in the VNNLIB header (lines 70–78), creating multi-megabyte property files.
• I/O overhead can be reduced by caching VNNLIB files when re-verifying identical (image, epsilon) pairs, rather than regenerating them on every call.
• Debugging workflow involves enabling debug logging, inspecting VNNLIB content, measuring actual verifier runtime, and profiling the property generator with cProfile.
• Performance optimization relies on parallel job execution, verifier-specific configuration files, and database caching of CompleteVerificationData results.

Frequently Asked Questions

Why does my verifier hang even with a short timeout?

The external verifier may ignore the timeout parameter or only support specific wall-clock arguments. Check the concrete implementation in autoverify.verifier.verifier.Verifier to verify timeout handling. Some verifiers require explicit configuration flags in the config file to enable timeout enforcement.

How can I reduce memory usage during verification?

Large memory footprints usually result from the property generator embedding full image tensors as CSV metadata in the VNNLIB header (lines 70–78 of verification_context.py). Reduce image resolution before verification or disable the metadata header if your use case permits. For ImageNet-scale tensors, this change can reduce property files from megabytes to kilobytes.

What timeout values should I use for different attack methods?

For FGSM attacks implemented in fgsm_attack.py, set timeout=30 seconds as these typically complete in under 5 seconds. For PGD attacks in pgd_attack.py, which perform iterative optimization, use 300–600 seconds to allow convergence. Always profile your specific network architecture, as deeper networks may require additional time.

How do I enable debug logging to trace timeout issues?

Configure the logger for ada_verona.verification_module.auto_verify_module to DEBUG level. This outputs the exact timeout= value passed to the verifier at lines 84–95 of auto_verify_module.py, allowing you to confirm the framework is correctly forwarding your specified limit to the external verification engine.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}