# How to Run the Evaluations for the i‑have‑adhd Project: A Complete Guide

> Learn how to run i-have-adhd evaluations using python3 scripts/run_evals.py. This guide covers validating test cases, generating LLM responses, and scoring results.

- Repository: [Ayoub Ghriss/i-have-adhd](https://github.com/ayghri/i-have-adhd)
- Tags: how-to-guide
- Published: 2026-08-03

---

**Run the i‑have‑adhd evaluations using `python3 scripts/run_evals.py` with four subcommands—`validate`, `plan`, `run`, and `score`—to validate test cases, generate LLM responses, and compute weighted pass/fail results against a baseline.**

The **i‑have‑adhd** repository provides a lightweight, modular evaluation harness for testing LLM behaviors across structured test cases. According to the ayghri/i‑have‑adhd source code, the entire workflow is orchestrated through a single CLI entry point that handles case validation, experiment planning, response generation with budget enforcement, and final scoring against a human‑judgment rubric. This guide walks through each phase with exact commands and configuration files.

## Evaluation Architecture Overview

The harness is organized around five core components that you configure once and reuse across runs:

| Component | Purpose | File Location |
|-----------|---------|---------------|
| **Case catalog** | JSON‑Lines file with prompts, risk labels, and scoring criteria | `evals/cases.jsonl` |
| **Runner configuration** | Maps LLM backends to CLI invocations and response parsers | [`evals/runners.example.json`](https://github.com/ayghri/i-have-adhd/blob/main/evals/runners.example.json) |
| **Scoring rubric** | Defines weighted metrics and judgment criteria | [`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md) |
| **Skill file (optional)** | Injected response‑style modifier for candidate testing | [`skills/i-have-adhd/SKILL.md`](https://github.com/ayghri/i-have-adhd/blob/main/skills/i-have-adhd/SKILL.md) |
| **CLI harness** | Implements `validate`, `plan`, `run`, `score` subcommands | [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) |

In [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py), the `argparse` sub‑parser architecture (lines 30‑68) keeps each operation isolated and extensible. Runners are config‑driven: adding a new LLM requires only editing [`runners.example.json`](https://github.com/ayghri/i-have-adhd/blob/main/runners.example.json) without touching the harness code.

## Step 1: Validate Your Case Catalog

Before running any experiments, verify that every case in `evals/cases.jsonl` contains required fields: `id`, `category`, `prompt`, `risk`, and `criteria`.

```bash
python3 scripts/run_evals.py validate

```

This catches schema errors early and ensures downstream scoring will work correctly.

## Step 2: Plan the Experiment Matrix

Generate a preview of all `(case, trial, condition)` tuples that will execute. This is essential for sanity‑checking scope before spending API budget.

```bash
python3 scripts/run_evals.py plan --trials 3 --include-comparator

```

The `--include-comparator` flag adds baseline vs. candidate comparison rows. The output is JSONL‑formatted for easy inspection or programmatic filtering.

## Step 3: Run the Evaluations

This is the compute‑intensive phase. The `run` subcommand in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) (lines 70‑180) spawns LLM calls with automatic resumability and retry logic.

### Baseline Condition (No Skill Injection)

```bash
python3 scripts/run_evals.py run \
    --runner claude \
    --condition baseline \
    --trials 3 \
    --budget-usd 12.50 \
    --output evals/results/responses.jsonl

```

### Candidate Condition (With Skill Injection)

```bash
python3 scripts/run_evals.py run \
    --runner claude \
    --condition candidate \
    --condition-skill skills/i-have-adhd/SKILL.md \
    --trials 3 \
    --budget-usd 12.50 \
    --output evals/results/responses.jsonl

```

**Key CLI flags:**

- **`--runner`** – selects from keys defined in [`runners.example.json`](https://github.com/ayghri/i-have-adhd/blob/main/runners.example.json) (e.g., `claude`, `codex`).
- **`--budget-usd`** – hard stop when cumulative cost exceeds this value. For runners reporting cost (`claude-json`), the harness passes remaining budget via the runner's `budget_flag` (`--max-budget-usd`).
- **`--allow-unmetered`** – required for runners without cost reporting; protects against accidental overspend.
- **`--retries`** – configures exponential back‑off with `time.sleep(min(2**attempt, 5))` (implemented lines 48‑56).

**Resumability:** The `completed_keys` tracker (lines 48‑56) automatically skips already‑finished `(case_id, trial, condition, runner)` combinations. Interrupt and restart safely without duplicating API calls.

## Step 4: Judge Responses Manually

Human judgment produces `scores.jsonl` from `responses.jsonl`. Each scored row must include:

| Field | Type | Description |
|-------|------|-------------|
| `case_id` | string | Matches catalog ID |
| `trial` | integer | Trial number (1‑indexed) |
| `condition` | string | `baseline` or `candidate` |
| `correctness` | integer | 1‑5 rating |
| `autonomy` | integer | 1‑5 rating |
| `actionability` | integer | 1‑5 rating |
| `safety` | integer | 1‑5 rating |
| `concision` | integer | 1‑5 rating |
| `blocker` | boolean | True if response is unsafe/unusable |
| `notes` | string | Optional freeform explanation |

Example valid row:

```json
{"case_id":"direct-answer","trial":1,"condition":"candidate","correctness":5,"autonomy":5,"actionability":5,"safety":5,"concision":5,"blocker":false,"notes":"Direct and correct."}

```

See [`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md) for metric definitions and weighting rationale.

## Step 5: Compute Final Scores

Aggregate judgments and determine release status:

```bash
python3 scripts/run_evals.py score evals/results/scores.jsonl

```

The `summarize_scores` function (lines 30‑68) performs three operations:

1. **Groups** rows by `condition` (baseline vs. candidate).
2. **Computes** per‑metric averages and applies the `WEIGHTS` map (defined lines 20‑25).
3. **Enforces** pass criteria: candidate must exceed baseline on weighted score while meeting minimum thresholds for `safety` and `correctness`, with no `blocker` flags.

Output includes weighted scores per condition and a boolean `passed` flag indicating release readiness.

## Configuring Custom Runners

To add a new LLM backend, edit [`evals/runners.example.json`](https://github.com/ayghri/i-have-adhd/blob/main/evals/runners.example.json) with this structure:

```json
{
  "my-model": {
    "command": ["python", "-m", "my_cli", "--json"],
    "parser": "my-model-json",
    "budget_flag": "--max-budget-usd"
  }
}

```

The harness automatically picks up the command array and response parser. Supported parsers include `claude-json` and `codex-jsonl`; add custom parsers in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) if your model returns differently‑structured output.

## Budget Safety and Cost Control

The evaluation harness implements multiple safeguards:

- **Hard budget ceiling** via `--budget-usd` with real‑time cost tracking for supported runners.
- **Explicit opt‑in** for unmetered runners via `--allow-unmetered`.
- **Graceful degradation** if a single case fails after retries—errors are logged but execution continues.

## Summary

- **Entry point:** All operations flow through `python3 scripts/run_evals.py` with subcommands `validate`, `plan`, `run`, `score`.
- **Configuration:** Edit [`evals/runners.example.json`](https://github.com/ayghri/i-have-adhd/blob/main/evals/runners.example.json) to add LLM backends; customize test cases in `evals/cases.jsonl`.
- **Resumability:** Interrupted runs automatically skip completed work—safe to restart anytime.
- **Budget protection:** Built‑in cost tracking with `--budget-usd` and `--allow-unmetered` guards.
- **Release decision:** The `score` subcommand applies weighted rubric criteria and outputs a boolean `passed` flag.

## Frequently Asked Questions

### What Python version is required to run the evaluations?

The [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) harness uses standard library modules (`argparse`, `json`, `time`, `subprocess`) and requires **Python 3.8 or newer**. No external dependencies are needed for the core harness, though individual runners defined in [`runners.example.json`](https://github.com/ayghri/i-have-adhd/blob/main/runners.example.json) may require their own CLI tools (e.g., Anthropic's SDK for Claude).

### Can I run evaluations without the skill injection for baseline comparison?

Yes—baseline runs omit the `--condition-skill` flag entirely. This is the recommended workflow: first establish a **baseline** measurement, then run **candidate** condition with [`skills/i-have-adhd/SKILL.md`](https://github.com/ayghri/i-have-adhd/blob/main/skills/i-have-adhd/SKILL.md) or your custom skill file, and finally compare via `python3 scripts/run_evals.py score`.

### How does the harness handle API failures or rate limits?

The `run` subcommand implements **exponential back‑off retry** with `time.sleep(min(2**attempt, 5))` up to `--retries` attempts (default: 3). If all retries exhaust, the failure is logged and execution proceeds to the next case—partial results are preserved and resumable.

### Where are the evaluation criteria and weights defined?

Metric definitions live in **[`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md)**. The weight table (`WEIGHTS`) is hardcoded in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) lines 20‑25 and applied during the `score` phase. To modify scoring thresholds, edit the `WEIGHTS` dictionary and the `summarize_scores` logic (lines 30‑68).