# How the i‑have‑ADHD Evaluation Framework Works: A Technical Guide to LLM Response Scoring

> Discover the technical i-have-ADHD evaluation framework. Learn how LLM responses are scored using five weighted metrics and automated gates to ensure quality and prevent regressions.

- Repository: [Ayoub Ghriss/i-have-adhd](https://github.com/ayghri/i-have-adhd)
- Tags: deep-dive
- Published: 2026-08-01

---

**The i‑have‑ADHD evaluation framework scores language‑model responses by comparing baseline and candidate conditions across five weighted metrics—correctness, autonomy, actionability, safety, and concision—with automated release gates blocking regressions.**

This open‑source evaluation system, developed in `ayghri/i-have-adhd`, provides a reproducible pipeline for assessing how different prompting strategies or model configurations affect response quality. The framework is designed for researchers and engineers who need rigorous, metrics‑driven comparisons before deploying LLM‑powered features.

## Overview of the Five Evaluation Metrics

Each response pair is scored on five dimensions with predefined weights defined in `run_evals.WEIGHTS`:

- **Correctness** – factual accuracy and alignment with ground truth
- **Autonomy** – appropriate level of independent action proposed
- **Actionability** – clarity and feasibility of recommended steps
- **Safety** – avoidance of harmful, risky, or inappropriate content
- **Concision** – appropriate brevity without omitting critical information

These metrics feed into a final `weighted_score` that determines whether a candidate condition passes the release gate.

## Case Catalog and Data Structure

All evaluation scenarios live in **`evals/cases.jsonl`**, a JSON‑Lines file where each case specifies:

```json
{
  "id": "unique-case-id",
  "category": "task-category",
  "prompt": "The user query or task description",
  "risk": "low|medium|high",
  "criteria": ["specific", "scoring", "criteria"]
}

```

The `risk` field is critical: high‑risk cases trigger stricter release gate thresholds. The `criteria` list provides rubric items that human reviewers use to assign scores consistently.

## Loading and Validating Cases

Before any evaluation runs, the framework validates data integrity through `load_cases()` and `validate_cases()` in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py).

**Validation checks include:**

- Required fields present (`id`, `category`, `prompt`, `risk`, `criteria`)
- Unique, non‑empty `id` values
- Valid `risk` levels (low/medium/high only)
- Non‑empty `criteria` arrays

Run validation standalone:

```bash
python -m scripts.run_evals validate

# → prints "Evaluation cases are valid." or a list of errors

```

## Running Evaluations: The CLI Orchestration

The `run` subcommand in **[`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py)** manages the full evaluation pipeline.

### Step‑by‑Step Execution

1. **Build condition prompts** via `_condition_prompt()`:
   - **Baseline**: raw task prompt
   - **Candidate**: task prompt prepended with skill file when `--condition-skill` is provided

2. **Launch runner subprocess** – supports Claude, Codex, or stub runners with `--budget-usd` caps

3. **Parse responses** via `_parse_response()` according to `response_format`:
   - `text`
   - `claude-json`
   - `codex-jsonl`

4. **Record trial** with fields: `case_id`, `trial`, `condition`, `runner`, `response`, `usage`, `cost_usd`

### Example: Full Evaluation Matrix

```bash

# Baseline condition

python -m scripts.run_evals run \
  --runner stub \
  --condition baseline \
  --output results/baseline.jsonl \
  --budget-usd 10

# Candidate condition with response style modification

python -m scripts.run_evals run \
  --runner stub \
  --condition candidate \
  --condition-skill response_style.txt \
  --output results/candidate.jsonl \
  --budget-usd 10

```

## Scoring and Release Gates

After human reviewers score response pairs, the `score` subcommand processes results through `summarize_scores()`.

### Score Processing Pipeline

1. **Validate score rows** with `_validate_score()`
2. **Check pairing integrity** via `_check_pairing()` – ensures baseline and candidate judged on identical case/trial combinations
3. **Compute per‑metric averages** and apply `WEIGHTS` to derive `weighted_score`
4. **Evaluate release gate** with three hard requirements:
   - No blocking findings attached to any score
   - Correctness regression ≤ 0.1 vs. baseline
   - Safety regression ≤ 0.1 vs. baseline
   - Weighted score exceeds baseline

### Score Command

```bash
python -m scripts.run_evals score results/scores.jsonl

# → prints JSON summary with weighted scores and release_gate status

```

**Sample output structure:**

```json
{
  "weights": {"correctness": 0.3, "autonomy": 0.2, "actionability": 0.2, "safety": 0.2, "concision": 0.1},
  "baseline": {"weighted_score": 3.8, "correctness": 4.0, ...},
  "candidate": {"weighted_score": 4.1, "correctness": 4.2, ...},
  "release_gate": {"passed": true, "reasons": []}
}

```

## Key Files and Implementation Details

| File | Purpose |
|------|---------|
| [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) | Core CLI, case loading, validation, runner orchestration, scoring logic |
| `evals/cases.jsonl` | Source‑of‑truth for all evaluation prompts and criteria |
| [`tests/test_run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/tests/test_run_evals.py) | Unit tests for validation, scoring, budget handling, pairing rules |

The framework's design prioritizes **reproducibility**: every trial records full provenance (runner, cost, raw response), and the pairing requirement guarantees fair A/B comparisons.

## Extending the Framework

Teams can adapt the i‑have‑ADHD evaluation framework by:

- Adding cases to `evals/cases.jsonl` with appropriate risk ratings
- Implementing custom runners that conform to the subprocess interface
- Adjusting `WEIGHTS` in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) to reflect product priorities
- Modifying release gate thresholds for different deployment stages

All modifications are test‑covered; run `pytest tests/test_run_evals.py` to verify changes don't break validation, scoring, or pairing logic.

## Summary

- **Five weighted metrics** (correctness, autonomy, actionability, safety, concision) form the core evaluation rubric
- **JSON‑Lines case catalog** at `evals/cases.jsonl` stores all evaluation scenarios with risk classifications
- **CLI-driven workflow** in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) handles validation, execution, and scoring
- **Strict pairing requirement** ensures baseline and candidate comparisons are statistically valid
- **Automated release gate** blocks deployment on correctness or safety regressions exceeding 0.1 points

## Frequently Asked Questions

### What file contains all the evaluation prompts for i‑have‑ADHD?

All evaluation prompts, criteria, and risk classifications are stored in **`evals/cases.jsonl`** as a JSON‑Lines file. Each line contains one case with fields for `id`, `category`, `prompt`, `risk`, and `criteria`.

### How does the framework ensure fair comparisons between baseline and candidate conditions?

The `_check_pairing()` function in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) validates that scoring rows for baseline and candidate conditions share identical `case_id` and `trial` combinations. The `summarize_scores()` function raises an error if any mismatched pairs are detected, preventing biased comparisons from uneven evaluation coverage.

### What triggers a release gate failure in the i‑have‑ADHD evaluation framework?

A candidate fails the release gate if any of these conditions occur: blocking findings attached to scores, correctness regression greater than 0.1 versus baseline, safety regression greater than 0.1 versus baseline, or a weighted score below the baseline value.

### Can I run evaluations with a budget cap to limit API costs?

Yes. The `--budget-usd` flag passed to `python -m scripts.run_evals run` enforces a spending limit. The runner subprocess terminates if projected costs exceed this threshold, with cost tracking recorded in each trial's `cost_usd` field.