# How i-have-adhd Caps Evaluation Lists: Budget-Controlled Execution

> Discover how i-have-adhd caps evaluation lists. Learn about the budget-cap mechanism in run_evals.py that controls spending and prevents exceeding monetary thresholds for efficient execution.

- Repository: [Ayoub Ghriss/i-have-adhd](https://github.com/ayghri/i-have-adhd)
- Tags: deep-dive
- Published: 2026-08-04

---

**The i-have-adhd repository enforces strict spending limits through a budget-cap mechanism in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) that tracks cumulative costs and halts execution when the monetary threshold is reached.**

The `ayghri/i-have-adhd` project implements rigorous financial safeguards to prevent runaway evaluation costs when testing language model skills. By capping evaluation lists through real-time budget monitoring, the system ensures that automated testing never exceeds user-defined spending limits. This article examines the specific implementation details of how i-have-adhd caps lists using the budget enforcement logic found in the evaluation runner script.

## Budget Validation and Default Limits

The capping mechanism begins with strict validation of the `--budget-usd` parameter. Before any evaluations run, [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) enforces a hard ceiling of **$25 USD** while requiring a positive value:

```python
if args.budget_usd <= 0 or args.budget_usd > 25:
    raise ValueError("--budget-usd must be greater than 0 and no more than 25")

```

This check appears at lines 35–36 of the script, ensuring that users cannot accidentally specify excessive spending limits or invalid negative values. The default value of $25 provides a reasonable baseline for evaluation runs while preventing unexpected bills.

## Real-Time Cost Accumulation

As the system processes each evaluation case, it maintains a running total of expenses through the `reported_cost` variable. After each runner invocation returns its cost data, the script accumulates the value:

```python
reported_cost += float(cost or 0)

```

Located at lines 89–90, this accumulator ensures that every completed evaluation contributes to the total spend calculation. By converting null values to zero with `float(cost or 0)`, the system gracefully handles runners that return no cost data without crashing the budget tracking logic.

## Runtime Termination on Budget Exhaustion

Before initiating each new trial, the script calculates the remaining budget and evaluates whether execution should continue. This is the core mechanism that actually caps the list of evaluations:

```python
remaining = args.budget_usd - reported_cost
if remaining <= 0:
    print("Budget exhausted; stopping.", file=sys.stderr)
    return 2

```

Found at lines 48–52, this check compares the accumulated `reported_cost` against the user-specified limit. When funds are depleted, the script writes a diagnostic message to standard error and exits with status code `2`, immediately stopping further evaluations regardless of how many cases remain in the queue.

## Passing Remaining Budget to Individual Runners

The system supports granular cost control by propagating the remaining budget to individual runner processes. When a runner's configuration includes a `budget_flag` key (typically defined in [`evals/runners.example.json`](https://github.com/ayghri/i-have-adhd/blob/main/evals/runners.example.json)), the remaining amount is appended to the command-line invocation:

```python
if runner.get("budget_flag"):
    invocation.extend([runner["budget_flag"], f"{remaining:.4f}"])

```

This logic at lines 54–56 allows downstream model providers to enforce their own internal limits based on the available funds. The formatting to four decimal places (`f"{remaining:.4f}"`) ensures precise monetary values are passed to the runner executable.

## Unmetered Mode and Silent Overspending Prevention

To prevent the system from capping lists based on inaccurate zero-cost assumptions, the script refuses to run with response formats that do not report dollar costs unless explicitly authorized. This safeguard prevents silent budget overruns:

```python
if response_format != "claude-json" and not args.allow_unmetered:
    raise RuntimeError(
        f"The {response_format!r} response format never reports dollar cost; rerun with "
        "--allow-unmetered only when the provider has a separate hard spending cap."
    )

```

Located at lines 21–25, this validation ensures that users must explicitly opt-in to `--allow-unmetered` mode when using response formats (such as plain text) that lack cost reporting. This forces acknowledgment that the provider must implement separate spending controls, as the i-have-adhd budget cap cannot function without accurate cost feedback.

## Summary

- **Hard limit validation**: The script enforces a maximum $25 USD budget with minimum values greater than zero at startup.
- **Continuous monitoring**: Each evaluation's cost is added to a running total stored in `reported_cost`.
- **Automatic termination**: Execution stops with exit code `2` when `remaining` budget reaches zero or below.
- **Runner-level control**: Remaining budget values propagate to individual runners via the `budget_flag` configuration option.
- **Explicit unmetered consent**: Users must pass `--allow-unmetered` to run evaluations with cost-oblivious response formats.

## Frequently Asked Questions

### What is the maximum budget cap allowed in i-have-adhd?

The system enforces a hard maximum of **$25 USD** for the `--budget-usd` parameter. Attempting to specify a value greater than 25 or less than or equal to zero triggers a `ValueError` at lines 35–36 of [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py).

### How does i-have-adhd handle runners that don't report costs?

By default, the script raises a `RuntimeError` and refuses to execute if the response format does not report dollar costs (anything other than `claude-json`). Users must explicitly add the `--allow-unmetered` flag to proceed, acknowledging that the provider implements separate spending caps.

### Can individual evaluation runners receive the remaining budget amount?

Yes. If the runner configuration in [`evals/runners.example.json`](https://github.com/ayghri/i-have-adhd/blob/main/evals/runners.example.json) defines a `budget_flag`, the script appends both the flag and the formatted remaining budget (to four decimal places) to the runner's command-line invocation at lines 54–56.

### What happens when the evaluation budget is exhausted mid-run?

The script prints "Budget exhausted; stopping." to standard error and returns exit code `2`, immediately terminating the evaluation loop regardless of pending cases. This ensures the system never exceeds the specified monetary cap.