Evaluation Framework and Test Case Structure in i-have-adhd: A Complete Technical Guide

The i-have-adhd repository provides a lightweight, four-stage evaluation harness—validate, plan, run, and score—driven by JSONL case catalogs and weighted scoring rubrics.

This open-source project, hosted at ayghri/i-have-adhd, ships a complete framework for systematically testing LLM behavior with ADHD-friendly response formatting. The evaluation system orchestrates test execution, captures model outputs, and applies rigorous pass/fail criteria to determine release readiness.

---

Core Architecture of the Evaluation Framework

The framework centers on scripts/run_evals.py, a unified CLI harness that coordinates four distinct workflow stages. All evaluation data lives under the evals/ directory with clear separation between case definitions, runner configurations, and scoring rules.

Four-Stage Workflow

Stage	Command	Purpose
Validate	run_evals.py validate	Checks evals/cases.jsonl for schema compliance via read_jsonl and validate_cases
Plan	run_evals.py plan	Generates a run matrix of (case_id, trial, condition) tuples
Run	run_evals.py run	Executes LLM calls, injects skill files for candidate conditions, streams results to JSONL
Score	run_evals.py score	Aggregates weighted metrics, enforces pairing constraints, computes release-gate verdict

Each stage supports resumability through a shared completed_keys utility that skips already-finished rows.

Required Data Files

File	Role
evals/cases.jsonl	Line-delimited catalog of evaluation cases (validated by validate_cases)
evals/rubric.md	Defines weighted metrics and release-gate thresholds
evals/runners.example.json	Maps LLM providers to CLI invocations and response formats

---

How Test Cases Are Defined

Test cases in i-have-adhd follow a strict JSON schema enforced during validation. Each line in evals/cases.jsonl represents one independent evaluation case.

Mandatory Case Fields

Key	Type	Validation Rule
id	str	Non-empty, unique across catalog
category	str	Classification bucket (e.g., direct-answer, debugging)
prompt	str	User-facing task fed to the LLM
risk	enum	Must be "low", "medium", or "high"
criteria	list[str]	Non-empty list of human-verification checkpoints

Example Test Case

{
  "id": "direct-answer",
  "category": "direct-answer",
  "prompt": "What is 17 multiplied by 6?",
  "risk": "low",
  "criteria": [
    "Answers 102.",
    "Does not invent unnecessary steps for the user."
  ]
}

The validate_cases function enforces all constraints: required field presence, id uniqueness, valid risk values, and non-empty criteria. When the run sub-command executes, the prompt field becomes the raw task, with _condition_prompt wrapping it in the skill file content for candidate or comparator conditions.

---

Scoring Mechanics and Release Gates

The scoring system implements weighted aggregation with strict regression checks, all defined in evals/rubric.md and implemented in summarize_scores.

Metric Weights

WEIGHTS = {
    "correctness": 0.35,
    "autonomy": 0.25,
    "actionability": 0.20,
    "safety": 0.10,
    "concision": 0.10,
}

Score Processing Pipeline

1. _validate_score – Ensures each row contains all five metrics, plus blocker (bool) and notes
2. Grouping by condition – Separates baseline and candidate results
3. _check_pairing – Verifies identical (case_id, trial) coverage across conditions
4. Aggregation – Computes mean per metric, then weighted sum yields weighted_score
5. Release-gate decision – Fails if any blocker=True, correctness/safety regresses >0.1 points, or candidate ≤ baseline

The final output includes original weights, per-condition aggregates, and a release_gate object with explicit pass/fail reasoning.

---

Practical CLI Examples

Validate Your Case Catalog

python3 scripts/run_evals.py validate

Returns case count and validation status or surfaces specific schema violations.

Generate a Run Matrix

python3 scripts/run_evals.py plan --trials 3 --include-comparator > matrix.jsonl

Produces rows like:

{"case_id":"direct-answer","trial":1,"condition":"baseline"}
{"case_id":"direct-answer","trial":1,"condition":"candidate"}
{"case_id":"direct-answer","trial":1,"condition":"comparator"}

Execute Evaluations

python3 scripts/run_evals.py run \
  --runner claude \
  --condition candidate \
  --condition-skill skills/i-have-adhd/SKILL.md \
  --output evals/results/responses.jsonl

Respects budget limits and retry policies; appends to output without re-running completed keys.

Score Completed Judgments

python3 scripts/run_evals.py score evals/results/scores.jsonl

Emits structured JSON with metrics, weighted scores, and release-gate verdict.

---

Summary

• Unified CLI: scripts/run_evals.py implements validate→plan→run→score workflow with resume support via completed_keys
• Strict schema: evals/cases.jsonl requires id, category, prompt, risk, and criteria fields, enforced by validate_cases
• Condition-aware execution: _condition_prompt injects skills/i-have-adhd/SKILL.md for candidate/comparator runs
• Weighted scoring: Five metrics with defined weights; summarize_scores enforces pairing and regression checks
• Automated gating: Release fails on blockers, safety/correctness regression >0.1, or insufficient improvement over baseline

---

Frequently Asked Questions

How do I add a new test case to the evaluation framework?

Create a JSON object with all five required fields (id, category, prompt, risk, criteria) and append it as a new line to evals/cases.jsonl. Run python3 scripts/run_evals.py validate to verify schema compliance before planning your evaluation matrix.

What is the difference between baseline, candidate, and comparator conditions?

Baseline runs the raw prompt without modifications. Candidate wraps the prompt with the ADHD skill file via _condition_prompt. Comparator (optional) tests an alternative skill or configuration for direct comparison. The scoring pipeline requires identical case/trial coverage across baseline and candidate.

Why does my scoring command fail with a pairing error?

The _check_pairing function enforces that every condition was judged on exactly the same set of (case_id, trial) pairs. Ensure your scores file contains matching rows for all conditions without omissions or duplicates, then retry.

Can I resume a long-running evaluation without losing progress?

Yes. All run_evals.py sub-commands use completed_keys to track finished rows. If your process interrupts, re-run the same command—the harness automatically skips already-completed cases and continues from where it stopped.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}