How i-have-adhd Caps Evaluation Lists: Budget-Controlled Execution

The i-have-adhd repository enforces strict spending limits through a budget-cap mechanism in scripts/run_evals.py that tracks cumulative costs and halts execution when the monetary threshold is reached.

The ayghri/i-have-adhd project implements rigorous financial safeguards to prevent runaway evaluation costs when testing language model skills. By capping evaluation lists through real-time budget monitoring, the system ensures that automated testing never exceeds user-defined spending limits. This article examines the specific implementation details of how i-have-adhd caps lists using the budget enforcement logic found in the evaluation runner script.

Budget Validation and Default Limits

The capping mechanism begins with strict validation of the --budget-usd parameter. Before any evaluations run, scripts/run_evals.py enforces a hard ceiling of $25 USD while requiring a positive value:

if args.budget_usd <= 0 or args.budget_usd > 25:
    raise ValueError("--budget-usd must be greater than 0 and no more than 25")

This check appears at lines 35–36 of the script, ensuring that users cannot accidentally specify excessive spending limits or invalid negative values. The default value of $25 provides a reasonable baseline for evaluation runs while preventing unexpected bills.

Real-Time Cost Accumulation

As the system processes each evaluation case, it maintains a running total of expenses through the reported_cost variable. After each runner invocation returns its cost data, the script accumulates the value:

reported_cost += float(cost or 0)

Located at lines 89–90, this accumulator ensures that every completed evaluation contributes to the total spend calculation. By converting null values to zero with float(cost or 0), the system gracefully handles runners that return no cost data without crashing the budget tracking logic.

Runtime Termination on Budget Exhaustion

Before initiating each new trial, the script calculates the remaining budget and evaluates whether execution should continue. This is the core mechanism that actually caps the list of evaluations:

remaining = args.budget_usd - reported_cost
if remaining <= 0:
    print("Budget exhausted; stopping.", file=sys.stderr)
    return 2

Found at lines 48–52, this check compares the accumulated reported_cost against the user-specified limit. When funds are depleted, the script writes a diagnostic message to standard error and exits with status code 2, immediately stopping further evaluations regardless of how many cases remain in the queue.

Passing Remaining Budget to Individual Runners

The system supports granular cost control by propagating the remaining budget to individual runner processes. When a runner's configuration includes a budget_flag key (typically defined in evals/runners.example.json), the remaining amount is appended to the command-line invocation:

if runner.get("budget_flag"):
    invocation.extend([runner["budget_flag"], f"{remaining:.4f}"])

This logic at lines 54–56 allows downstream model providers to enforce their own internal limits based on the available funds. The formatting to four decimal places (f"{remaining:.4f}") ensures precise monetary values are passed to the runner executable.

Unmetered Mode and Silent Overspending Prevention

To prevent the system from capping lists based on inaccurate zero-cost assumptions, the script refuses to run with response formats that do not report dollar costs unless explicitly authorized. This safeguard prevents silent budget overruns:

if response_format != "claude-json" and not args.allow_unmetered:
    raise RuntimeError(
        f"The {response_format!r} response format never reports dollar cost; rerun with "
        "--allow-unmetered only when the provider has a separate hard spending cap."
    )

Located at lines 21–25, this validation ensures that users must explicitly opt-in to --allow-unmetered mode when using response formats (such as plain text) that lack cost reporting. This forces acknowledgment that the provider must implement separate spending controls, as the i-have-adhd budget cap cannot function without accurate cost feedback.

Summary

• Hard limit validation: The script enforces a maximum $25 USD budget with minimum values greater than zero at startup.
• Continuous monitoring: Each evaluation's cost is added to a running total stored in reported_cost.
• Automatic termination: Execution stops with exit code 2 when remaining budget reaches zero or below.
• Runner-level control: Remaining budget values propagate to individual runners via the budget_flag configuration option.
• Explicit unmetered consent: Users must pass --allow-unmetered to run evaluations with cost-oblivious response formats.

Frequently Asked Questions

What is the maximum budget cap allowed in i-have-adhd?

The system enforces a hard maximum of $25 USD for the --budget-usd parameter. Attempting to specify a value greater than 25 or less than or equal to zero triggers a ValueError at lines 35–36 of scripts/run_evals.py.

How does i-have-adhd handle runners that don't report costs?

By default, the script raises a RuntimeError and refuses to execute if the response format does not report dollar costs (anything other than claude-json). Users must explicitly add the --allow-unmetered flag to proceed, acknowledging that the provider implements separate spending caps.

Can individual evaluation runners receive the remaining budget amount?

Yes. If the runner configuration in evals/runners.example.json defines a budget_flag, the script appends both the flag and the formatted remaining budget (to four decimal places) to the runner's command-line invocation at lines 54–56.

What happens when the evaluation budget is exhausted mid-run?

The script prints "Budget exhausted; stopping." to standard error and returns exit code 2, immediately terminating the evaluation loop regardless of pending cases. This ensures the system never exceeds the specified monetary cap.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}