# Evaluation Framework and Test Case Structure in i-have-adhd: A Complete Technical Guide

> Explore the i-have-adhd evaluation framework and test case structure. Learn how validate, plan, run, and score stages use JSONL catalogs and weighted rubrics for robust assessment.

- Repository: [Ayoub Ghriss/i-have-adhd](https://github.com/ayghri/i-have-adhd)
- Tags: deep-dive
- Published: 2026-08-20

---

**The i-have-adhd repository provides a lightweight, four-stage evaluation harness—validate, plan, run, and score—driven by JSONL case catalogs and weighted scoring rubrics.**

This open-source project, hosted at `ayghri/i-have-adhd`, ships a complete framework for systematically testing LLM behavior with ADHD-friendly response formatting. The evaluation system orchestrates test execution, captures model outputs, and applies rigorous pass/fail criteria to determine release readiness.

---

## Core Architecture of the Evaluation Framework

The framework centers on **[`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py)**, a unified CLI harness that coordinates four distinct workflow stages. All evaluation data lives under the **`evals/`** directory with clear separation between case definitions, runner configurations, and scoring rules.

### Four-Stage Workflow

| Stage | Command | Purpose |
|-------|---------|---------|
| **Validate** | `run_evals.py validate` | Checks `evals/cases.jsonl` for schema compliance via `read_jsonl` and `validate_cases` |
| **Plan** | `run_evals.py plan` | Generates a run matrix of `(case_id, trial, condition)` tuples |
| **Run** | `run_evals.py run` | Executes LLM calls, injects skill files for candidate conditions, streams results to JSONL |
| **Score** | `run_evals.py score` | Aggregates weighted metrics, enforces pairing constraints, computes release-gate verdict |

Each stage supports resumability through a shared `completed_keys` utility that skips already-finished rows.

### Required Data Files

| File | Role |
|------|------|
| `evals/cases.jsonl` | Line-delimited catalog of evaluation cases (validated by `validate_cases`) |
| [`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md) | Defines weighted metrics and release-gate thresholds |
| [`evals/runners.example.json`](https://github.com/ayghri/i-have-adhd/blob/main/evals/runners.example.json) | Maps LLM providers to CLI invocations and response formats |

---

## How Test Cases Are Defined

Test cases in i-have-adhd follow a strict JSON schema enforced during validation. Each line in `evals/cases.jsonl` represents one independent evaluation case.

### Mandatory Case Fields

| Key | Type | Validation Rule |
|-----|------|---------------|
| `id` | `str` | Non-empty, unique across catalog |
| `category` | `str` | Classification bucket (e.g., `direct-answer`, `debugging`) |
| `prompt` | `str` | User-facing task fed to the LLM |
| `risk` | enum | Must be `"low"`, `"medium"`, or `"high"` |
| `criteria` | `list[str]` | Non-empty list of human-verification checkpoints |

### Example Test Case

```json
{
  "id": "direct-answer",
  "category": "direct-answer",
  "prompt": "What is 17 multiplied by 6?",
  "risk": "low",
  "criteria": [
    "Answers 102.",
    "Does not invent unnecessary steps for the user."
  ]
}

```

The `validate_cases` function enforces all constraints: required field presence, `id` uniqueness, valid `risk` values, and non-empty `criteria`. When the `run` sub-command executes, the `prompt` field becomes the raw task, with `_condition_prompt` wrapping it in the skill file content for `candidate` or `comparator` conditions.

---

## Scoring Mechanics and Release Gates

The scoring system implements weighted aggregation with strict regression checks, all defined in [`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md) and implemented in `summarize_scores`.

### Metric Weights

```python
WEIGHTS = {
    "correctness": 0.35,
    "autonomy": 0.25,
    "actionability": 0.20,
    "safety": 0.10,
    "concision": 0.10,
}

```

### Score Processing Pipeline

1. **_validate_score** – Ensures each row contains all five metrics, plus `blocker` (bool) and `notes`
2. **Grouping by condition** – Separates baseline and candidate results
3. **_check_pairing** – Verifies identical `(case_id, trial)` coverage across conditions
4. **Aggregation** – Computes mean per metric, then weighted sum yields `weighted_score`
5. **Release-gate decision** – Fails if any `blocker=True`, correctness/safety regresses >0.1 points, or candidate ≤ baseline

The final output includes original weights, per-condition aggregates, and a `release_gate` object with explicit pass/fail reasoning.

---

## Practical CLI Examples

### Validate Your Case Catalog

```bash
python3 scripts/run_evals.py validate

```

Returns case count and validation status or surfaces specific schema violations.

### Generate a Run Matrix

```bash
python3 scripts/run_evals.py plan --trials 3 --include-comparator > matrix.jsonl

```

Produces rows like:

```json
{"case_id":"direct-answer","trial":1,"condition":"baseline"}
{"case_id":"direct-answer","trial":1,"condition":"candidate"}
{"case_id":"direct-answer","trial":1,"condition":"comparator"}

```

### Execute Evaluations

```bash
python3 scripts/run_evals.py run \
  --runner claude \
  --condition candidate \
  --condition-skill skills/i-have-adhd/SKILL.md \
  --output evals/results/responses.jsonl

```

Respects budget limits and retry policies; appends to output without re-running completed keys.

### Score Completed Judgments

```bash
python3 scripts/run_evals.py score evals/results/scores.jsonl

```

Emits structured JSON with metrics, weighted scores, and release-gate verdict.

---

## Summary

- **Unified CLI**: [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) implements validate→plan→run→score workflow with resume support via `completed_keys`
- **Strict schema**: `evals/cases.jsonl` requires `id`, `category`, `prompt`, `risk`, and `criteria` fields, enforced by `validate_cases`
- **Condition-aware execution**: `_condition_prompt` injects [`skills/i-have-adhd/SKILL.md`](https://github.com/ayghri/i-have-adhd/blob/main/skills/i-have-adhd/SKILL.md) for candidate/comparator runs
- **Weighted scoring**: Five metrics with defined weights; `summarize_scores` enforces pairing and regression checks
- **Automated gating**: Release fails on blockers, safety/correctness regression >0.1, or insufficient improvement over baseline

---

## Frequently Asked Questions

### How do I add a new test case to the evaluation framework?

Create a JSON object with all five required fields (`id`, `category`, `prompt`, `risk`, `criteria`) and append it as a new line to `evals/cases.jsonl`. Run `python3 scripts/run_evals.py validate` to verify schema compliance before planning your evaluation matrix.

### What is the difference between baseline, candidate, and comparator conditions?

**Baseline** runs the raw prompt without modifications. **Candidate** wraps the prompt with the ADHD skill file via `_condition_prompt`. **Comparator** (optional) tests an alternative skill or configuration for direct comparison. The scoring pipeline requires identical case/trial coverage across baseline and candidate.

### Why does my scoring command fail with a pairing error?

The `_check_pairing` function enforces that every condition was judged on exactly the same set of `(case_id, trial)` pairs. Ensure your scores file contains matching rows for all conditions without omissions or duplicates, then retry.

### Can I resume a long-running evaluation without losing progress?

Yes. All [`run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/run_evals.py) sub-commands use `completed_keys` to track finished rows. If your process interrupts, re-run the same command—the harness automatically skips already-completed cases and continues from where it stopped.