How to Run the Evaluations for the i‑have‑adhd Project: A Complete Guide

Run the i‑have‑adhd evaluations using python3 scripts/run_evals.py with four subcommands—validate, plan, run, and score—to validate test cases, generate LLM responses, and compute weighted pass/fail results against a baseline.

The i‑have‑adhd repository provides a lightweight, modular evaluation harness for testing LLM behaviors across structured test cases. According to the ayghri/i‑have‑adhd source code, the entire workflow is orchestrated through a single CLI entry point that handles case validation, experiment planning, response generation with budget enforcement, and final scoring against a human‑judgment rubric. This guide walks through each phase with exact commands and configuration files.

Evaluation Architecture Overview

The harness is organized around five core components that you configure once and reuse across runs:

Component	Purpose	File Location
Case catalog	JSON‑Lines file with prompts, risk labels, and scoring criteria	evals/cases.jsonl
Runner configuration	Maps LLM backends to CLI invocations and response parsers	evals/runners.example.json
Scoring rubric	Defines weighted metrics and judgment criteria	evals/rubric.md
Skill file (optional)	Injected response‑style modifier for candidate testing	skills/i-have-adhd/SKILL.md
CLI harness	Implements validate, plan, run, score subcommands	scripts/run_evals.py

In scripts/run_evals.py, the argparse sub‑parser architecture (lines 30‑68) keeps each operation isolated and extensible. Runners are config‑driven: adding a new LLM requires only editing runners.example.json without touching the harness code.

Step 1: Validate Your Case Catalog

Before running any experiments, verify that every case in evals/cases.jsonl contains required fields: id, category, prompt, risk, and criteria.

python3 scripts/run_evals.py validate

This catches schema errors early and ensures downstream scoring will work correctly.

Step 2: Plan the Experiment Matrix

Generate a preview of all (case, trial, condition) tuples that will execute. This is essential for sanity‑checking scope before spending API budget.

python3 scripts/run_evals.py plan --trials 3 --include-comparator

The --include-comparator flag adds baseline vs. candidate comparison rows. The output is JSONL‑formatted for easy inspection or programmatic filtering.

Step 3: Run the Evaluations

This is the compute‑intensive phase. The run subcommand in scripts/run_evals.py (lines 70‑180) spawns LLM calls with automatic resumability and retry logic.

Baseline Condition (No Skill Injection)

python3 scripts/run_evals.py run \
    --runner claude \
    --condition baseline \
    --trials 3 \
    --budget-usd 12.50 \
    --output evals/results/responses.jsonl

Candidate Condition (With Skill Injection)

python3 scripts/run_evals.py run \
    --runner claude \
    --condition candidate \
    --condition-skill skills/i-have-adhd/SKILL.md \
    --trials 3 \
    --budget-usd 12.50 \
    --output evals/results/responses.jsonl

Key CLI flags:

• --runner – selects from keys defined in runners.example.json (e.g., claude, codex).
• --budget-usd – hard stop when cumulative cost exceeds this value. For runners reporting cost (claude-json), the harness passes remaining budget via the runner's budget_flag (--max-budget-usd).
• --allow-unmetered – required for runners without cost reporting; protects against accidental overspend.
• --retries – configures exponential back‑off with time.sleep(min(2**attempt, 5)) (implemented lines 48‑56).

Resumability: The completed_keys tracker (lines 48‑56) automatically skips already‑finished (case_id, trial, condition, runner) combinations. Interrupt and restart safely without duplicating API calls.

Step 4: Judge Responses Manually

Human judgment produces scores.jsonl from responses.jsonl. Each scored row must include:

Field	Type	Description
case_id	string	Matches catalog ID
trial	integer	Trial number (1‑indexed)
condition	string	baseline or candidate
correctness	integer	1‑5 rating
autonomy	integer	1‑5 rating
actionability	integer	1‑5 rating
safety	integer	1‑5 rating
concision	integer	1‑5 rating
blocker	boolean	True if response is unsafe/unusable
notes	string	Optional freeform explanation

Example valid row:

{"case_id":"direct-answer","trial":1,"condition":"candidate","correctness":5,"autonomy":5,"actionability":5,"safety":5,"concision":5,"blocker":false,"notes":"Direct and correct."}

See evals/rubric.md for metric definitions and weighting rationale.

Step 5: Compute Final Scores

Aggregate judgments and determine release status:

python3 scripts/run_evals.py score evals/results/scores.jsonl

The summarize_scores function (lines 30‑68) performs three operations:

1. Groups rows by condition (baseline vs. candidate).
2. Computes per‑metric averages and applies the WEIGHTS map (defined lines 20‑25).
3. Enforces pass criteria: candidate must exceed baseline on weighted score while meeting minimum thresholds for safety and correctness, with no blocker flags.

Output includes weighted scores per condition and a boolean passed flag indicating release readiness.

Configuring Custom Runners

To add a new LLM backend, edit evals/runners.example.json with this structure:

{
  "my-model": {
    "command": ["python", "-m", "my_cli", "--json"],
    "parser": "my-model-json",
    "budget_flag": "--max-budget-usd"
  }
}

The harness automatically picks up the command array and response parser. Supported parsers include claude-json and codex-jsonl; add custom parsers in scripts/run_evals.py if your model returns differently‑structured output.

Budget Safety and Cost Control

The evaluation harness implements multiple safeguards:

• Hard budget ceiling via --budget-usd with real‑time cost tracking for supported runners.
• Explicit opt‑in for unmetered runners via --allow-unmetered.
• Graceful degradation if a single case fails after retries—errors are logged but execution continues.

Summary

• Entry point: All operations flow through python3 scripts/run_evals.py with subcommands validate, plan, run, score.
• Configuration: Edit evals/runners.example.json to add LLM backends; customize test cases in evals/cases.jsonl.
• Resumability: Interrupted runs automatically skip completed work—safe to restart anytime.
• Budget protection: Built‑in cost tracking with --budget-usd and --allow-unmetered guards.
• Release decision: The score subcommand applies weighted rubric criteria and outputs a boolean passed flag.

Frequently Asked Questions

What Python version is required to run the evaluations?

The scripts/run_evals.py harness uses standard library modules (argparse, json, time, subprocess) and requires Python 3.8 or newer. No external dependencies are needed for the core harness, though individual runners defined in runners.example.json may require their own CLI tools (e.g., Anthropic's SDK for Claude).

Can I run evaluations without the skill injection for baseline comparison?

Yes—baseline runs omit the --condition-skill flag entirely. This is the recommended workflow: first establish a baseline measurement, then run candidate condition with skills/i-have-adhd/SKILL.md or your custom skill file, and finally compare via python3 scripts/run_evals.py score.

How does the harness handle API failures or rate limits?

The run subcommand implements exponential back‑off retry with time.sleep(min(2**attempt, 5)) up to --retries attempts (default: 3). If all retries exhaust, the failure is logged and execution proceeds to the next case—partial results are preserved and resumable.

Where are the evaluation criteria and weights defined?

Metric definitions live in evals/rubric.md. The weight table (WEIGHTS) is hardcoded in scripts/run_evals.py lines 20‑25 and applied during the score phase. To modify scoring thresholds, edit the WEIGHTS dictionary and the summarize_scores logic (lines 30‑68).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}