# Debugging Verification Timeouts and Performance Issues in VERONA

> Troubleshoot VERONA verification timeouts and performance bottlenecks. Learn to identify common issues like incorrect timeout values, large property headers, and blocking I/O.

- Repository: [ADA research/verona](https://github.com/ada-research/verona)
- Tags: debugging
- Published: 2026-02-23

---

**Most verification timeouts in VERONA originate from three sources: the timeout value passed to external verifiers, oversized VNNLIB property headers containing full image CSV metadata, and blocking I/O during property serialization.**

VERONA, developed by ada-research, is an open-source framework designed for scalable neural network verification. When running robustness analysis on high-dimensional inputs or large datasets, developers frequently encounter premature timeouts, excessive memory consumption, and verifier hangs. Understanding how the `AutoVerifyModule` propagates timeout limits and manages property generation is essential for diagnosing and resolving these performance bottlenecks.

## How VERONA Applies Verification Timeouts

VERONA’s verification workflow relies on three core abstractions that handle timeout management:

- **Verification Context** ([`ada_verona/database/verification_context.py`](https://github.com/ada-research/verona/blob/main/ada_verona/database/verification_context.py)) – Bundles the neural network, data point, and property generator.
- **Verification Module** ([`ada_verona/verification_module/verification_module.py`](https://github.com/ada-research/verona/blob/main/ada_verona/verification_module/verification_module.py)) – Defines the abstract `verify` API that concrete verifiers must implement.
- **AutoVerifyModule** ([`ada_verona/verification_module/auto_verify_module.py`](https://github.com/ada-research/verona/blob/main/ada_verona/verification_module/auto_verify_module.py)) – Drives external verifiers like SDP-CROWN or Marabou and manages timeout enforcement, configuration files, and result parsing.

### Timeout Propagation Flow

The timeout mechanism operates across three distinct phases:

1. **Construction** – The caller supplies a **`timeout`** value in seconds when instantiating `AutoVerifyModule`. This value is stored in `self.timeout` at line 46 of [`auto_verify_module.py`](https://github.com/ada-research/verona/blob/main/auto_verify_module.py).

2. **Invocation** – When the `verify` method is called, the module forwards the timeout to the underlying verifier’s `verify_property` method (lines 84–95).

3. **Verifier Execution** – The external verifier (implemented in the *autoverify* package) receives the timeout and aborts the process if the wall-clock limit is reached.

Because the timeout is a simple float passed through these layers, most performance-related bugs stem from configuration errors rather than framework logic.

## Common Performance Bottlenecks

When debugging verification timeouts, identify your specific symptom to locate the root cause:

**Symptom: Verifier hangs for minutes despite a small timeout value**

- **Likely Cause:** The external verifier ignores the timeout parameter. Some verifiers only support wall-clock arguments or require specific configuration flags.
- **Where to Look:** Inspect the concrete verifier class in `autoverify.verifier.verifier.Verifier` to confirm timeout handling.

**Symptom: Large memory usage before timeout occurs**

- **Likely Cause:** The property generator embeds the full image as CSV data in the VNNLIB header (lines 70–78 of [`verification_context.py`](https://github.com/ada-research/verona/blob/main/verification_context.py)). For large images, this header can grow to several megabytes.
- **Where to Look:** Examine the generated `.vnnlib` file size and content.

**Symptom: Frequent "Error during verification" log messages**

- **Likely Cause:** `AutoVerifyModule` catches `Err` results and returns the error string (lines 108–110). The underlying `result.unwrap_err()` often contains the original timeout exception masked as a generic error.
- **Where to Look:** Enable debug logging to capture the raw exception from the verifier.

**Symptom: Slow performance during repeated verification calls**

- **Likely Cause:** `AutoVerifyModule` recreates the VNNLIB file on every `verify` call, causing significant I/O overhead when the same `(image, epsilon)` pair appears frequently.
- **Where to Look:** Profile file system operations during batch verification runs.

## Debugging Verification Timeouts Step-by-Step

Follow this systematic approach to isolate timeout and performance issues:

### Verify Timeout Propagation

Enable debug logging to confirm the timeout value reaches the verifier:

```python
import logging
logging.getLogger('ada_verona.verification_module.auto_verify_module').setLevel(logging.DEBUG)

```

The debug output will display the exact `timeout=` argument passed at lines 84–95 of [`auto_verify_module.py`](https://github.com/ada-research/verona/blob/main/auto_verify_module.py).

### Inspect the Generated VNNLIB File

Check for oversized metadata headers that slow down parsing:

```python
ctx.save_vnnlib_property(vnnlib)  # inside verify()

with open(vnnlib.path) as f:
    print(f.read()[:500])   # header + first 10 constraints

```

If the header contains a massive CSV list (lines 71–78), reduce image resolution or disable metadata embedding via feature flags.

### Measure Actual Verifier Runtime

Compare elapsed time against the configured timeout to detect non-compliance:

```python
import time
start = time.time()
result = self.verifier.verify_property(..., timeout=self.timeout)
print("Verifier elapsed:", time.time() - start)

```

A large discrepancy between elapsed time and `self.timeout` indicates the verifier is not respecting the limit.

### Tune Timeout Values by Attack Type

Adjust timeouts based on the verification algorithm:

- **FGSM attacks** ([`fgsm_attack.py`](https://github.com/ada-research/verona/blob/main/fgsm_attack.py)): Typically complete in under 5 seconds; set `timeout=30`.
- **PGD attacks** ([`pgd_attack.py`](https://github.com/ada-research/verona/blob/main/pgd_attack.py)): Iterative nature requires 300–600 seconds for convergence.

### Profile Property Generation

If CPU usage spikes before verification begins, profile the property generator:

```python
from ada_verona.verification_module.property_generator.property_generator import PropertyGenerator
import cProfile, pstats

pr = cProfile.Profile()
pr.enable()
vnn = pg.create_vnnlib_property(image, label, epsilon)
pr.disable()
pstats.Stats(pr).sort_stats('cumtime').print_stats(10)

```

## Architectural Levers for Better Performance

Optimize throughput using these framework features:

**Batch Verification**

`AutoVerifyModule` processes single data points. For large datasets, use the multiple-jobs scripts in `examples/scripts/multiple_jobs/` to launch parallel `AutoVerifyModule` instances.

**Verifier Configuration**

Pass a `config` file (see `examples/scripts/create_robustness_dist_*.py`) to adjust verifier-specific parameters like `max_iterations` or `use_gpu` for GPU-accelerated solving.

**Result Caching**

Store `CompleteVerificationData` objects in the database ([`ada_verona/database/verification_result.py`](https://github.com/ada-research/verona/blob/main/ada_verona/database/verification_result.py)) to avoid re-running verification on identical epsilons.

**Parallel Property Generation**

The property generator is pure Python. Wrap calls in `concurrent.futures.ThreadPoolExecutor` if property creation dominates runtime during batch processing.

## Practical Code Examples

### Running Verification with a Custom Timeout

```python
from pathlib import Path
from ada_verona.verification_module.auto_verify_module import AutoVerifyModule
from ada_verona.database.verification_context import VerificationContext
from autoverify.verifier.verifier import VerifierFactory

# Load a verifier (e.g. SDP-CROWN)

verifier = VerifierFactory.create("sdpcrown")

# Increase timeout to 10 minutes for complex properties

auto_mod = AutoVerifyModule(verifier=verifier, timeout=600)

# Build verification context

ctx = VerificationContext.load(
    network_path=Path("models/resnet18.onnx"),
    dataset_path=Path("data/mnist_test.pt"),
    property_generator="one2one"
)

# Verify at epsilon = 0.02

result = auto_mod.verify(ctx, epsilon=0.02)
print(result)   # → CompleteVerificationData or error string

```

### Detecting Timeout-Related Errors

```python
from ada_verona.verification_module.auto_verify_module import AutoVerifyModule
import logging

logging.basicConfig(level=logging.INFO)

result = auto_mod.verify(ctx, epsilon=0.05)
if isinstance(result, str) and "timeout" in result.lower():
    print("Verification hit the timeout limit – consider raising it.")

```

### Profiling Property Generation

```python
import cProfile, pstats
from ada_verona.verification_module.property_generator.one2one_property_generator import One2OnePropertyGenerator
from ada_verona.database.verification_context import VerificationContext

pg = One2OnePropertyGenerator()
ctx = VerificationContext.load(...)
image = ctx.data_point.data.reshape(-1).detach().numpy()

pr = cProfile.Profile()
pr.enable()
_ = pg.create_vnnlib_property(image, ctx.data_point.label, epsilon=0.01)
pr.disable()
pstats.Stats(pr).sort_stats('cumtime').print_stats(5)

```

## Summary

- **Timeout propagation** flows from `AutoVerifyModule` construction (line 46) through to the external verifier’s `verify_property` method (lines 84–95), but external verifiers may not always respect the wall-clock limit.
- **Memory bottlenecks** typically occur when [`verification_context.py`](https://github.com/ada-research/verona/blob/main/verification_context.py) embeds full image data as CSV in the VNNLIB header (lines 70–78), creating multi-megabyte property files.
- **I/O overhead** can be reduced by caching VNNLIB files when re-verifying identical `(image, epsilon)` pairs, rather than regenerating them on every call.
- **Debugging workflow** involves enabling debug logging, inspecting VNNLIB content, measuring actual verifier runtime, and profiling the property generator with `cProfile`.
- **Performance optimization** relies on parallel job execution, verifier-specific configuration files, and database caching of `CompleteVerificationData` results.

## Frequently Asked Questions

### Why does my verifier hang even with a short timeout?

The external verifier may ignore the timeout parameter or only support specific wall-clock arguments. Check the concrete implementation in `autoverify.verifier.verifier.Verifier` to verify timeout handling. Some verifiers require explicit configuration flags in the config file to enable timeout enforcement.

### How can I reduce memory usage during verification?

Large memory footprints usually result from the property generator embedding full image tensors as CSV metadata in the VNNLIB header (lines 70–78 of [`verification_context.py`](https://github.com/ada-research/verona/blob/main/verification_context.py)). Reduce image resolution before verification or disable the metadata header if your use case permits. For ImageNet-scale tensors, this change can reduce property files from megabytes to kilobytes.

### What timeout values should I use for different attack methods?

For **FGSM attacks** implemented in [`fgsm_attack.py`](https://github.com/ada-research/verona/blob/main/fgsm_attack.py), set `timeout=30` seconds as these typically complete in under 5 seconds. For **PGD attacks** in [`pgd_attack.py`](https://github.com/ada-research/verona/blob/main/pgd_attack.py), which perform iterative optimization, use 300–600 seconds to allow convergence. Always profile your specific network architecture, as deeper networks may require additional time.

### How do I enable debug logging to trace timeout issues?

Configure the logger for `ada_verona.verification_module.auto_verify_module` to `DEBUG` level. This outputs the exact `timeout=` value passed to the verifier at lines 84–95 of [`auto_verify_module.py`](https://github.com/ada-research/verona/blob/main/auto_verify_module.py), allowing you to confirm the framework is correctly forwarding your specified limit to the external verification engine.