How to Run the Evaluation Framework for i-have-adhd: Complete CLI Guide

To run the evaluation framework for i-have-adhd, execute python3 scripts/run_evals.py with sub-commands validate, plan, run, or score—driven by evals/cases.jsonl, a runner configuration file, and your LLM provider of choice.

The i-have-adhd evaluation harness is a self-contained Python tool that validates case catalogs, executes controlled experiments across baseline and candidate conditions, and computes weighted release-gate scores. According to the source code in ayghri/i-have-adhd, the framework enforces budget limits, ensures paired trial integrity, and produces machine-readable results for manual judgment.

---

Core Architecture of the Evaluation Harness

The evaluation framework lives in scripts/run_evals.py and exposes four CLI sub-commands: validate, plan, run, and score. Understanding these components helps you run the i-have-adhd evaluation framework correctly.

Key Functions

• run_evaluations() – Orchestrates case loading from evals/cases.jsonl, applies per-run budget enforcement, constructs prompts, and persists results to JSON-L files.
• _condition_prompt() – Injects skill instructions from a SKILL.md file into prompts for non-baseline conditions.
• summarize_scores() – Aggregates manually judged scores, applies the weighting table from evals/rubric.md, and renders the final pass/fail verdict.

The CLI parser at the bottom of scripts/run_evals.py defines all arguments and sub-commands, making the tool entirely self-contained with no external services beyond your chosen LLM provider.

---

Prerequisites and Configuration Files

Before running any evaluations, you need three files in place:

1. evals/cases.jsonl – JSON-L catalog of evaluation cases with fields: id, prompt, risk_level, and evaluation criteria.
2. evals/runners.example.json – Runner configuration specifying CLI commands and budget flags for each supported LLM.
3. skills/i-have-adhd/SKILL.md – The response-style skill file injected during candidate condition runs.

Copy and modify evals/runners.example.json to create your own runners.json. The example file includes configurations for Claude and Codex with --max-budget-usd flags.

---

Step 1: Validate the Case Catalog

Run the validation sub-command to check evals/cases.jsonl for required fields, unique IDs, and valid risk levels:

python3 scripts/run_evals.py validate

This catches schema errors before you spend budget on doomed runs. The validator is implemented in the validate sub-command handler in scripts/run_evals.py.

---

Step 2: Preview the Run Matrix (Optional)

Use the plan sub-command to see exactly which (case, trial, condition) tuples will execute:

python3 scripts/run_evals.py plan --trials 3 --include-comparator

Output is JSON-L to stdout, making it easy to inspect or pipe to other tools. This step costs nothing—no LLM calls are made.

---

Step 3: Run Baseline Evaluations

Execute the baseline condition (no skill injection) to establish performance without the i-have-adhd modifications:

python3 scripts/run_evals.py run \
  --runner-config evals/runners.example.json \
  --runner claude \
  --condition baseline \
  --trials 3 \
  --budget-usd 12.5 \
  --output evals/results/baseline.jsonl

Key parameters:

• --runner-config – Path to your runner configuration JSON.
• --runner – Which runner key to use from that configuration.
• --condition – Must be baseline, candidate, or comparator.
• --trials – Number of independent executions per case.
• --budget-usd – Per-run budget cap (maximum $25 USD).
• --output – Destination for JSON-L results.

The budget enforcement logic (lines 35–51 of run_evaluations) stops execution when remaining budget would be exceeded.

---

Step 4: Run Candidate Evaluations

Execute the candidate condition with the i-have-adhd skill injected into every prompt:

python3 scripts/run_evals.py run \
  --runner-config evals/runners.example.json \
  --runner claude \
  --condition candidate \
  --condition-skill skills/i-have-adhd/SKILL.md \
  --trials 3 \
  --budget-usd 12.5 \
  --output evals/results/candidate.jsonl

The --condition-skill flag triggers _condition_prompt() to prepend the skill instructions to each task prompt. Use the same --trials value as baseline to preserve paired comparison integrity.

---

Step 5: Optional Comparator Evaluations

Add a third skill as a reference point:

python3 scripts/run_evals.py run \
  --runner-config evals/runners.example.json \
  --runner claude \
  --condition comparator \
  --condition-skill path/to/other/skill.md \
  --trials 3 \
  --budget-usd 12.5 \
  --output evals/results/comparator.jsonl

This enables three-way analysis: baseline vs. your candidate vs. an alternative implementation.

---

Step 6: Score the Results

After manual judgment, compute aggregate metrics and the release-gate verdict:

python3 scripts/run_evals.py score evals/results/scores.jsonl

Creating the Scores File

1. Open your response files (evals/results/baseline.jsonl, evals/results/candidate.jsonl).
2. For each response, create a judgment object with these exact fields:

{"case_id":"direct-answer","trial":1,"condition":"candidate","correctness":5,"autonomy":5,"actionability":5,"safety":5,"concision":5,"blocker":false,"notes":"Direct and correct."}

Required fields:

• case_id, trial, condition – Must match the original run.
• correctness, autonomy, actionability, safety, concision – Integer scores (typically 1–5).
• blocker – Boolean indicating a safety-critical failure.
• notes – Free-form justification.

3. Save all judgments to evals/results/scores.jsonl.
4. Run the score command.

The scoring logic (summarize_scores, lines 30–68) validates row schemas via _validate_score(), checks baseline/candidate pairing via _check_pairing(), and produces:

• Per-condition aggregated metrics.
• Weighted composite scores from WEIGHTS in evals/rubric.md.
• release_gate object with passed boolean and failure reasons (e.g., correctness regression, safety blocker).

---

Complete Evaluation Workflow

# Validate cases

python3 scripts/run_evals.py validate

# Preview matrix

python3 scripts/run_evals.py plan --trials 3

# Run baseline

python3 scripts/run_evals.py run \
  --runner-config evals/runners.example.json \
  --runner claude \
  --condition baseline \
  --trials 3 \
  --budget-usd 12.5 \
  --output evals/results/baseline.jsonl

# Run candidate with i-have-adhd skill

python3 scripts/run_evals.py run \
  --runner-config evals/runners.example.json \
  --runner claude \
  --condition candidate \
  --condition-skill skills/i-have-adhd/SKILL.md \
  --trials 3 \
  --budget-usd 12.5 \
  --output evals/results/candidate.jsonl

# After manual judgment:

python3 scripts/run_evals.py score evals/results/scores.jsonl

---

Key Reference Files

File	Purpose
scripts/run_evals.py	Core harness with run_evaluations(), _condition_prompt(), summarize_scores()
evals/README.md	Official evaluation documentation
evals/rubric.md	WEIGHTS table and scoring criteria
evals/cases.jsonl	Case catalog with prompts and risk levels
evals/runners.example.json	Example configurations for Claude/Codex
skills/i-have-adhd/SKILL.md	Skill instructions injected into candidate prompts

---

Summary

• The evaluation framework for i-have-adhd is a single Python script (scripts/run_evals.py) with four sub-commands: validate, plan, run, and score.
• Budget enforcement is built-in—set --budget-usd up to $25 per run and the harness stops before overspending.
• Paired comparison integrity is guaranteed by using identical --trials values across conditions and verified by _check_pairing() during scoring.
• Manual judgment is required between run and score—add weighted scores and blocker flags to create scores.jsonl.
• Release gate output indicates pass/fail with specific regression reasons, computed from WEIGHTS in evals/rubric.md.

---

Frequently Asked Questions

What LLM providers does the i-have-adhd evaluation framework support?

The framework supports any provider you can configure in a runner JSON file. The provided evals/runners.example.json includes working configurations for Claude (Anthropic) and Codex (OpenAI). Add new providers by specifying their CLI command and budget flag format.

Why does my run stop before completing all trials?

The budget enforcement in run_evaluations() stops execution when the remaining budget would be exhausted by the next case-trial pair. Increase --budget-usd (maximum $25) or reduce --trials to fit your estimation. Check lines 35–51 of run_evaluations for the exact calculation logic.

How do I ensure my candidate and baseline are comparable?

Use identical values for --trials and ensure both runs cover the same case_id values from evals/cases.jsonl. The score sub-command runs _check_pairing() to verify this matching and fails if cases or trials are mismatched between conditions.

Where are the weighted scoring rules defined?

The WEIGHTS table lives in evals/rubric.md. The summarize_scores() function in scripts/run_evals.py applies these weights to your manual judgments when computing final metrics and the release-gate verdict.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}