# How to Run the Evaluation Framework for i-have-adhd: Complete CLI Guide

> Learn to run the evaluation framework for i-have-adhd with this complete CLI guide. Execute python3 scripts/run_evals.py using validate plan run or score commands for LLM evaluation.

- Repository: [Ayoub Ghriss/i-have-adhd](https://github.com/ayghri/i-have-adhd)
- Tags: how-to-guide
- Published: 2026-09-01

---

**To run the evaluation framework for i-have-adhd, execute `python3 scripts/run_evals.py` with sub-commands `validate`, `plan`, `run`, or `score`—driven by `evals/cases.jsonl`, a runner configuration file, and your LLM provider of choice.**

The **i-have-adhd** evaluation harness is a self-contained Python tool that validates case catalogs, executes controlled experiments across baseline and candidate conditions, and computes weighted release-gate scores. According to the source code in `ayghri/i-have-adhd`, the framework enforces budget limits, ensures paired trial integrity, and produces machine-readable results for manual judgment.

---

## Core Architecture of the Evaluation Harness

The evaluation framework lives in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) and exposes four CLI sub-commands: `validate`, `plan`, `run`, and `score`. Understanding these components helps you run the i-have-adhd evaluation framework correctly.

### Key Functions

- **`run_evaluations()`** – Orchestrates case loading from `evals/cases.jsonl`, applies per-run budget enforcement, constructs prompts, and persists results to JSON-L files.
- **`_condition_prompt()`** – Injects skill instructions from a [`SKILL.md`](https://github.com/ayghri/i-have-adhd/blob/main/SKILL.md) file into prompts for non-baseline conditions.
- **`summarize_scores()`** – Aggregates manually judged scores, applies the weighting table from [`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md), and renders the final pass/fail verdict.

The CLI parser at the bottom of [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) defines all arguments and sub-commands, making the tool entirely self-contained with no external services beyond your chosen LLM provider.

---

## Prerequisites and Configuration Files

Before running any evaluations, you need three files in place:

1. **`evals/cases.jsonl`** – JSON-L catalog of evaluation cases with fields: `id`, `prompt`, `risk_level`, and evaluation criteria.
2. **[`evals/runners.example.json`](https://github.com/ayghri/i-have-adhd/blob/main/evals/runners.example.json)** – Runner configuration specifying CLI commands and budget flags for each supported LLM.
3. **[`skills/i-have-adhd/SKILL.md`](https://github.com/ayghri/i-have-adhd/blob/main/skills/i-have-adhd/SKILL.md)** – The response-style skill file injected during candidate condition runs.

Copy and modify [`evals/runners.example.json`](https://github.com/ayghri/i-have-adhd/blob/main/evals/runners.example.json) to create your own [`runners.json`](https://github.com/ayghri/i-have-adhd/blob/main/runners.json). The example file includes configurations for **Claude** and **Codex** with `--max-budget-usd` flags.

---

## Step 1: Validate the Case Catalog

Run the validation sub-command to check `evals/cases.jsonl` for required fields, unique IDs, and valid risk levels:

```bash
python3 scripts/run_evals.py validate

```

This catches schema errors before you spend budget on doomed runs. The validator is implemented in the `validate` sub-command handler in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py).

---

## Step 2: Preview the Run Matrix (Optional)

Use the `plan` sub-command to see exactly which `(case, trial, condition)` tuples will execute:

```bash
python3 scripts/run_evals.py plan --trials 3 --include-comparator

```

Output is JSON-L to stdout, making it easy to inspect or pipe to other tools. This step costs nothing—no LLM calls are made.

---

## Step 3: Run Baseline Evaluations

Execute the baseline condition (no skill injection) to establish performance without the i-have-adhd modifications:

```bash
python3 scripts/run_evals.py run \
  --runner-config evals/runners.example.json \
  --runner claude \
  --condition baseline \
  --trials 3 \
  --budget-usd 12.5 \
  --output evals/results/baseline.jsonl

```

**Key parameters:**
- `--runner-config` – Path to your runner configuration JSON.
- `--runner` – Which runner key to use from that configuration.
- `--condition` – Must be `baseline`, `candidate`, or `comparator`.
- `--trials` – Number of independent executions per case.
- `--budget-usd` – Per-run budget cap (maximum $25 USD).
- `--output` – Destination for JSON-L results.

The budget enforcement logic (lines 35–51 of `run_evaluations`) stops execution when remaining budget would be exceeded.

---

## Step 4: Run Candidate Evaluations

Execute the candidate condition with the i-have-adhd skill injected into every prompt:

```bash
python3 scripts/run_evals.py run \
  --runner-config evals/runners.example.json \
  --runner claude \
  --condition candidate \
  --condition-skill skills/i-have-adhd/SKILL.md \
  --trials 3 \
  --budget-usd 12.5 \
  --output evals/results/candidate.jsonl

```

The `--condition-skill` flag triggers `_condition_prompt()` to prepend the skill instructions to each task prompt. Use the **same** `--trials` value as baseline to preserve paired comparison integrity.

---

## Step 5: Optional Comparator Evaluations

Add a third skill as a reference point:

```bash
python3 scripts/run_evals.py run \
  --runner-config evals/runners.example.json \
  --runner claude \
  --condition comparator \
  --condition-skill path/to/other/skill.md \
  --trials 3 \
  --budget-usd 12.5 \
  --output evals/results/comparator.jsonl

```

This enables three-way analysis: baseline vs. your candidate vs. an alternative implementation.

---

## Step 6: Score the Results

After manual judgment, compute aggregate metrics and the release-gate verdict:

```bash
python3 scripts/run_evals.py score evals/results/scores.jsonl

```

### Creating the Scores File

1. Open your response files (`evals/results/baseline.jsonl`, `evals/results/candidate.jsonl`).
2. For each response, create a judgment object with these exact fields:

```json
{"case_id":"direct-answer","trial":1,"condition":"candidate","correctness":5,"autonomy":5,"actionability":5,"safety":5,"concision":5,"blocker":false,"notes":"Direct and correct."}

```

**Required fields:**
- `case_id`, `trial`, `condition` – Must match the original run.
- `correctness`, `autonomy`, `actionability`, `safety`, `concision` – Integer scores (typically 1–5).
- `blocker` – Boolean indicating a safety-critical failure.
- `notes` – Free-form justification.

3. Save all judgments to `evals/results/scores.jsonl`.
4. Run the `score` command.

The scoring logic (`summarize_scores`, lines 30–68) validates row schemas via `_validate_score()`, checks baseline/candidate pairing via `_check_pairing()`, and produces:

- Per-condition aggregated metrics.
- Weighted composite scores from `WEIGHTS` in [`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md).
- `release_gate` object with `passed` boolean and failure reasons (e.g., correctness regression, safety blocker).

---

## Complete Evaluation Workflow

```bash

# Validate cases

python3 scripts/run_evals.py validate

# Preview matrix

python3 scripts/run_evals.py plan --trials 3

# Run baseline

python3 scripts/run_evals.py run \
  --runner-config evals/runners.example.json \
  --runner claude \
  --condition baseline \
  --trials 3 \
  --budget-usd 12.5 \
  --output evals/results/baseline.jsonl

# Run candidate with i-have-adhd skill

python3 scripts/run_evals.py run \
  --runner-config evals/runners.example.json \
  --runner claude \
  --condition candidate \
  --condition-skill skills/i-have-adhd/SKILL.md \
  --trials 3 \
  --budget-usd 12.5 \
  --output evals/results/candidate.jsonl

# After manual judgment:

python3 scripts/run_evals.py score evals/results/scores.jsonl

```

---

## Key Reference Files

| File | Purpose |
|------|---------|
| [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) | Core harness with `run_evaluations()`, `_condition_prompt()`, `summarize_scores()` |
| [`evals/README.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/README.md) | Official evaluation documentation |
| [`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md) | `WEIGHTS` table and scoring criteria |
| `evals/cases.jsonl` | Case catalog with prompts and risk levels |
| [`evals/runners.example.json`](https://github.com/ayghri/i-have-adhd/blob/main/evals/runners.example.json) | Example configurations for Claude/Codex |
| [`skills/i-have-adhd/SKILL.md`](https://github.com/ayghri/i-have-adhd/blob/main/skills/i-have-adhd/SKILL.md) | Skill instructions injected into candidate prompts |

---

## Summary

- **The evaluation framework for i-have-adhd** is a single Python script ([`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py)) with four sub-commands: `validate`, `plan`, `run`, and `score`.
- **Budget enforcement** is built-in—set `--budget-usd` up to $25 per run and the harness stops before overspending.
- **Paired comparison integrity** is guaranteed by using identical `--trials` values across conditions and verified by `_check_pairing()` during scoring.
- **Manual judgment** is required between `run` and `score`—add weighted scores and blocker flags to create `scores.jsonl`.
- **Release gate** output indicates pass/fail with specific regression reasons, computed from `WEIGHTS` in [`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md).

---

## Frequently Asked Questions

### What LLM providers does the i-have-adhd evaluation framework support?

The framework supports any provider you can configure in a runner JSON file. The provided [`evals/runners.example.json`](https://github.com/ayghri/i-have-adhd/blob/main/evals/runners.example.json) includes working configurations for **Claude** (Anthropic) and **Codex** (OpenAI). Add new providers by specifying their CLI command and budget flag format.

### Why does my run stop before completing all trials?

The budget enforcement in `run_evaluations()` stops execution when the remaining budget would be exhausted by the next case-trial pair. Increase `--budget-usd` (maximum $25) or reduce `--trials` to fit your estimation. Check lines 35–51 of `run_evaluations` for the exact calculation logic.

### How do I ensure my candidate and baseline are comparable?

Use identical values for `--trials` and ensure both runs cover the same `case_id` values from `evals/cases.jsonl`. The `score` sub-command runs `_check_pairing()` to verify this matching and fails if cases or trials are mismatched between conditions.

### Where are the weighted scoring rules defined?

The `WEIGHTS` table lives in [`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md). The `summarize_scores()` function in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) applies these weights to your manual judgments when computing final metrics and the release-gate verdict.