How the i‑have‑ADHD Evaluation Framework Works: A Technical Guide to LLM Response Scoring

The i‑have‑ADHD evaluation framework scores language‑model responses by comparing baseline and candidate conditions across five weighted metrics—correctness, autonomy, actionability, safety, and concision—with automated release gates blocking regressions.

This open‑source evaluation system, developed in ayghri/i-have-adhd, provides a reproducible pipeline for assessing how different prompting strategies or model configurations affect response quality. The framework is designed for researchers and engineers who need rigorous, metrics‑driven comparisons before deploying LLM‑powered features.

Overview of the Five Evaluation Metrics

Each response pair is scored on five dimensions with predefined weights defined in run_evals.WEIGHTS:

• Correctness – factual accuracy and alignment with ground truth
• Autonomy – appropriate level of independent action proposed
• Actionability – clarity and feasibility of recommended steps
• Safety – avoidance of harmful, risky, or inappropriate content
• Concision – appropriate brevity without omitting critical information

These metrics feed into a final weighted_score that determines whether a candidate condition passes the release gate.

Case Catalog and Data Structure

All evaluation scenarios live in evals/cases.jsonl, a JSON‑Lines file where each case specifies:

{
  "id": "unique-case-id",
  "category": "task-category",
  "prompt": "The user query or task description",
  "risk": "low|medium|high",
  "criteria": ["specific", "scoring", "criteria"]
}

The risk field is critical: high‑risk cases trigger stricter release gate thresholds. The criteria list provides rubric items that human reviewers use to assign scores consistently.

Loading and Validating Cases

Before any evaluation runs, the framework validates data integrity through load_cases() and validate_cases() in scripts/run_evals.py.

Validation checks include:

• Required fields present (id, category, prompt, risk, criteria)
• Unique, non‑empty id values
• Valid risk levels (low/medium/high only)
• Non‑empty criteria arrays

Run validation standalone:

python -m scripts.run_evals validate

# → prints "Evaluation cases are valid." or a list of errors

Running Evaluations: The CLI Orchestration

The run subcommand in scripts/run_evals.py manages the full evaluation pipeline.

Step‑by‑Step Execution

1. Build condition prompts via _condition_prompt():

   • Baseline: raw task prompt
   • Candidate: task prompt prepended with skill file when --condition-skill is provided

2. Launch runner subprocess – supports Claude, Codex, or stub runners with --budget-usd caps

3. Parse responses via _parse_response() according to response_format:

   • text
   • claude-json
   • codex-jsonl

4. Record trial with fields: case_id, trial, condition, runner, response, usage, cost_usd

Example: Full Evaluation Matrix

# Baseline condition

python -m scripts.run_evals run \
  --runner stub \
  --condition baseline \
  --output results/baseline.jsonl \
  --budget-usd 10

# Candidate condition with response style modification

python -m scripts.run_evals run \
  --runner stub \
  --condition candidate \
  --condition-skill response_style.txt \
  --output results/candidate.jsonl \
  --budget-usd 10

Scoring and Release Gates

After human reviewers score response pairs, the score subcommand processes results through summarize_scores().

Score Processing Pipeline

1. Validate score rows with _validate_score()
2. Check pairing integrity via _check_pairing() – ensures baseline and candidate judged on identical case/trial combinations
3. Compute per‑metric averages and apply WEIGHTS to derive weighted_score
4. Evaluate release gate with three hard requirements:
   • No blocking findings attached to any score
   • Correctness regression ≤ 0.1 vs. baseline
   • Safety regression ≤ 0.1 vs. baseline
   • Weighted score exceeds baseline

Score Command

python -m scripts.run_evals score results/scores.jsonl

# → prints JSON summary with weighted scores and release_gate status

Sample output structure:

{
  "weights": {"correctness": 0.3, "autonomy": 0.2, "actionability": 0.2, "safety": 0.2, "concision": 0.1},
  "baseline": {"weighted_score": 3.8, "correctness": 4.0, ...},
  "candidate": {"weighted_score": 4.1, "correctness": 4.2, ...},
  "release_gate": {"passed": true, "reasons": []}
}

Key Files and Implementation Details

File	Purpose
scripts/run_evals.py	Core CLI, case loading, validation, runner orchestration, scoring logic
evals/cases.jsonl	Source‑of‑truth for all evaluation prompts and criteria
tests/test_run_evals.py	Unit tests for validation, scoring, budget handling, pairing rules

The framework's design prioritizes reproducibility: every trial records full provenance (runner, cost, raw response), and the pairing requirement guarantees fair A/B comparisons.

Extending the Framework

Teams can adapt the i‑have‑ADHD evaluation framework by:

• Adding cases to evals/cases.jsonl with appropriate risk ratings
• Implementing custom runners that conform to the subprocess interface
• Adjusting WEIGHTS in scripts/run_evals.py to reflect product priorities
• Modifying release gate thresholds for different deployment stages

All modifications are test‑covered; run pytest tests/test_run_evals.py to verify changes don't break validation, scoring, or pairing logic.

Summary

• Five weighted metrics (correctness, autonomy, actionability, safety, concision) form the core evaluation rubric
• JSON‑Lines case catalog at evals/cases.jsonl stores all evaluation scenarios with risk classifications
• CLI-driven workflow in scripts/run_evals.py handles validation, execution, and scoring
• Strict pairing requirement ensures baseline and candidate comparisons are statistically valid
• Automated release gate blocks deployment on correctness or safety regressions exceeding 0.1 points

Frequently Asked Questions

What file contains all the evaluation prompts for i‑have‑ADHD?

All evaluation prompts, criteria, and risk classifications are stored in evals/cases.jsonl as a JSON‑Lines file. Each line contains one case with fields for id, category, prompt, risk, and criteria.

How does the framework ensure fair comparisons between baseline and candidate conditions?

The _check_pairing() function in scripts/run_evals.py validates that scoring rows for baseline and candidate conditions share identical case_id and trial combinations. The summarize_scores() function raises an error if any mismatched pairs are detected, preventing biased comparisons from uneven evaluation coverage.

What triggers a release gate failure in the i‑have‑ADHD evaluation framework?

A candidate fails the release gate if any of these conditions occur: blocking findings attached to scores, correctness regression greater than 0.1 versus baseline, safety regression greater than 0.1 versus baseline, or a weighted score below the baseline value.

Can I run evaluations with a budget cap to limit API costs?

Yes. The --budget-usd flag passed to python -m scripts.run_evals run enforces a spending limit. The runner subprocess terminates if projected costs exceed this threshold, with cost tracking recorded in each trial's cost_usd field.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}