# What Data Is Used for Evaluating the i‑Have‑ADHD Skill? A Deep Dive into the Evaluation Framework

> Discover what data evaluates the i-have-adhd skill. Learn about the framework using test cases from evals/cases.jsonl and human scores across five metrics and three risk tiers.

- Repository: [Ayoub Ghriss/i-have-adhd](https://github.com/ayghri/i-have-adhd)
- Tags: deep-dive
- Published: 2026-08-03

---

**The i‑have‑adHD skill is evaluated using a curated catalog of test cases stored in `evals/cases.jsonl`, paired with human‑produced score rows that measure five metrics across three risk tiers.**

The `ayghri/i‑have‑adhd` repository implements a rigorous, data‑driven evaluation pipeline to assess skill performance before release. Understanding this evaluation data structure is essential for contributors who want to add test cases, interpret results, or modify the scoring logic.

## The Evaluation Data: Two Core Components

The evaluation system relies on two distinct data sources that work together to validate skill behavior.

### 1. The Case Catalog (`evals/cases.jsonl`)

Each line in `evals/cases.jsonl` is a **JSON object** representing a single evaluation case with five required fields. The `validate_cases()` function in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) enforces this schema:

| Field | Purpose | Constraints |
|-------|---------|-------------|
| **`id`** | Unique case identifier | Must be unique across all cases |
| **`category`** | Functional grouping | Examples: `direct‑answer`, `coding`, `safety` |
| **`prompt`** | User query sent to the skill | Arbitrary text string |
| **`risk`** | Risk classification | Must be `low`, `medium`, or `high` |
| **`criteria`** | Success rubric items | Non‑empty list of requirement strings |

The validation logic in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) (lines 59–79) rejects any case missing fields, with duplicate IDs, or with invalid risk levels. High‑risk cases receive stricter judgment criteria during human evaluation.

### 2. Human‑Produced Score Rows

After running a case through the skill, human judges produce a **score row** with quantitative assessments. According to the source code in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) (lines 82–95), each score row must contain:

- `case_id` — links back to the case catalog
- `trial` and `condition` — tracks experimental runs (`baseline` vs. `candidate`)
- Five metric scores: **`correctness`**, **`autonomy`**, **`actionability`**, **`safety`**, **`concision`**
- `blocker` — boolean flag for critical failures
- `notes` — free‑form qualitative feedback

## Loading and Validating Evaluation Cases

The [`run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/run_evals.py) script provides `load_cases()` and `validate_cases()` functions to handle the case catalog. Here's how to use them programmatically:

```python
from pathlib import Path
from scripts.run_evals import load_cases, validate_cases

cases_path = Path("evals/cases.jsonl")

# Load the case catalog

cases = load_cases(cases_path)  # internally calls read_jsonl()

# Validate schema compliance

errors = validate_cases(cases)
if errors:
    raise ValueError(f"Validation failed:\n" + "\n".join(errors))

print(f"Loaded {len(cases)} valid evaluation cases")

```

The `read_jsonl()` utility (lines 44–45) handles line‑delimited JSON parsing, returning a list of dictionaries for downstream processing.

## Running Evaluations and Aggregating Scores

The evaluation pipeline executes cases through a configured LLM runner and collects human judgments. Use the CLI to execute a full evaluation run:

```bash
python -m scripts.run_evals run \
  --runner stub \
  --condition candidate \
  --output results/candidate.jsonl \
  --budget-usd 5.0 \
  --allow-unmetered

```

After scoring, aggregate results with `summarize_scores()`:

```python
from pathlib import Path
from scripts.run_evals import read_jsonl, summarize_scores

scores_path = Path("results/scores.jsonl")
scores = read_jsonl(scores_path)

summary = summarize_scores(scores)

# Access condition‑level results

baseline = summary["conditions"]["baseline"]["weighted_score"]
candidate = summary["conditions"]["candidate"]["weighted_score"]

print(f"Baseline: {baseline:.3f} | Candidate: {candidate:.3f}")
print(f"Release gate passed: {summary['release_gate']['passed']}")

```

## The Release Gate Decision Logic

The `summarize_scores()` function in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) (lines 30–68) implements a **weighted scoring algorithm** that determines release readiness. The process:

1. **Applies metric weights** — defined in the `WEIGHTS` map within [`run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/run_evals.py)
2. **Computes per‑condition averages** — aggregates across all cases and trials
3. **Evaluates three release criteria**:
   - No `blocker` flags in any scored case
   - No regression exceeding 0.1 points from baseline
   - Candidate weighted score strictly greater than baseline

Failure on any criterion prevents automatic release approval.

## Key Files in the Evaluation System

| File | Role |
|------|------|
| `evals/cases.jsonl` | Master catalog of evaluation prompts, risk tiers, and success criteria |
| [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) | Core engine: validation, LLM invocation, response parsing, score aggregation |
| [`tests/test_run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/tests/test_run_evals.py) | Unit tests for validation logic, summarization, and budget enforcement |
| [`evals/runners.example.json`](https://github.com/ayghri/i-have-adhd/blob/main/evals/runners.example.json) | Example configuration for LLM runner backends |

## Summary

- **Evaluation data for the i‑have‑adHD skill** splits into structured **case definitions** (`evals/cases.jsonl`) and **quantitative score rows** from human judges.
- **Five required fields** per case (`id`, `category`, `prompt`, `risk`, `criteria`) are strictly validated by `validate_cases()` in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py).
- **Five metrics** (`correctness`, `autonomy`, `actionability`, `safety`, `concision`) plus blocker flags feed into a **weighted release gate** that compares candidate against baseline performance.
- The [`run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/run_evals.py) module serves as the central orchestrator for loading, validating, executing, and summarizing all evaluation data.

## Frequently Asked Questions

### How do I add a new evaluation case to the i‑have‑adHD skill?

Append a JSON object to `evals/cases.jsonl` with all five required fields. Run `load_cases()` and `validate_cases()` from [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) to verify your addition passes schema validation before committing.

### What happens if a case has an invalid risk level?

The `validate_cases()` function rejects any case where `risk` is not exactly `low`, `medium`, or `high`. The error is returned in the validation errors list, and the case will not be included in evaluation runs.

### Can I modify the weights used in the release gate decision?

Yes. The `WEIGHTS` dictionary is defined in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py). Adjust these values to change how each metric contributes to the final weighted score, then re‑run `summarize_scores()` to see updated results.

### Where are evaluation results stored during a run?

The `--output` flag for the `run` command specifies the destination path (typically `results/<condition>.jsonl`). These files contain raw score rows before aggregation by `summarize_scores()`.