# How the i-have-adhd Skill Is Evaluated: A Complete Technical Guide > Learn how the i-have-adhd skill is evaluated via a three-stage pipeline: test case validation, LLM execution comparison, and weighted rubric scoring for release readiness. - Repository: [Ayoub Ghriss/i-have-adhd](https://github.com/ayghri/i-have-adhd) - Tags: deep-dive - Published: 2026-08-03 --- **The i-have-adhd skill is evaluated through a three-stage pipeline that validates test cases, executes LLM runs with and without the skill injected, and scores responses against a weighted rubric to determine release readiness.** The `ayghri/i-have-adhd` repository implements a rigorous evaluation harness to objectively measure whether the ADHD-specific response skill improves LLM outputs without compromising safety or correctness. Understanding how the i-have-adhd skill is evaluated requires examining the orchestration logic in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py), the case catalog structure, and the scoring methodology that gates releases. ## The Three-Stage Evaluation Pipeline The evaluation process follows a structured workflow defined in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py), moving from case validation through execution to final scoring. ### Stage 1: Validation and Planning Before any LLM calls occur, the harness validates the case catalog stored in `evals/cases.jsonl`. The `validate_cases` sub-command checks for required fields, uniqueness constraints, and proper risk categorization across scenarios like "direct-answer", "multi-step-progress", and "medical-boundary". The `plan` sub-command then generates a run matrix that pairs every validated case with two conditions: **baseline** (no skill) and **candidate** (with the i-have-adhd skill). This matrix ensures systematic A/B testing across the entire scenario catalog. ### Stage 2: Execution and Skill Injection The `run_evaluations` function executes the planned matrix, launching the selected LLM runner (e.g., Claude) for each case-trial pair. The critical logic resides in `_condition_prompt`, which handles conditional prompt construction: - **Baseline runs**: Receive only the raw task prompt from the case definition. - **Candidate runs**: Inject the skill file content into the prompt via a `` wrapper, forcing the model to obey the i-have-adhd response rules while answering the original task. The skill content is read from [`skills/i-have-adhd/SKILL.md`](https://github.com/ayghri/i-have-adhd/blob/main/skills/i-have-adhd/SKILL.md) and wrapped in XML-style tags before the task prompt. Results including response text, token usage, and cost are written to JSON-L files for downstream analysis. ### Stage 3: Scoring and Release-Gate Analysis Human judges score each response using the rubric defined in [`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md). The `summarize_scores` function aggregates these scores across five weighted metrics: correctness, autonomy, actionability, safety, and concision. The release gate applies strict pass/fail logic: - The candidate must not regress on **safety** or **correctness** compared to baseline. - No blocking findings can be present in the evaluation. - The candidate must exceed the baseline's weighted aggregate score. Only when all criteria are met does the harness set `passed: true` in the final JSON summary. ## Core Evaluation Concepts ### Case Catalog Structure The `evals/cases.jsonl` file contains the test scenarios that exercise the skill across diverse interaction patterns. Each JSON object defines the prompt context, risk level, and evaluation criteria. Cases span from simple direct answers to complex multi-step planning scenarios, ensuring the skill performs robustly across conversation types. ### Skill Injection via response_style The evaluation isolates the skill's impact by strictly controlling when rules apply. During candidate runs, [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) reads [`skills/i-have-adhd/SKILL.md`](https://github.com/ayghri/i-have-adhd/blob/main/skills/i-have-adhd/SKILL.md) and injects it into the prompt structure: ```python # Conceptual injection pattern used by _condition_prompt [Content of skills/i-have-adhd/SKILL.md] [Original task prompt from case] ``` This injection method ensures the model receives explicit formatting instructions without altering the underlying task difficulty, creating a fair comparison against baseline performance. ### The Five-Metric Scoring Rubric Scores are assigned on a 1-5 scale across five dimensions defined in [`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md): - **Correctness**: Factual accuracy and relevance to the query. - **Autonomy**: Appropriate level of independence vs. requesting clarification. - **Actionability**: Concrete, implementable guidance provided. - **Safety**: Absence of harmful content and proper handling of sensitive topics. - **Concision**: Efficiency of communication without excessive verbosity. The `WEIGHTS` mapping in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) applies these priorities during aggregation, with safety and correctness typically carrying higher weight in the final calculation. ### Release Gate Criteria The release gate prevents regression by enforcing minimum performance thresholds. The harness compares candidate scores against baseline results, checking for statistically significant improvements without trade-offs in critical dimensions. Failure reasons are explicitly logged, such as "Candidate has blocking safety or correctness findings," enabling targeted skill refinement. ## Running the Evaluation Harness Execute the full evaluation pipeline using the CLI commands defined in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py): ```bash # Step 1: Validate the case catalog integrity python3 scripts/run_evals.py validate # Step 2: Generate the run matrix with baseline and candidate conditions python3 scripts/run_evals.py plan --trials 3 --include-comparator > run_matrix.jsonl # Step 3: Execute baseline condition (no skill injection) python3 scripts/run_evals.py run \ --runner claude \ --condition baseline \ --trials 3 \ --budget-usd 12.50 \ --output evals/results/baseline.jsonl # Step 4: Execute candidate condition with i-have-adhd skill injected python3 scripts/run_evals.py run \ --runner claude \ --condition candidate \ --condition-skill skills/i-have-adhd/SKILL.md \ --trials 3 \ --budget-usd 12.50 \ --output evals/results/candidate.jsonl # Step 5: Score results and check release gate python3 scripts/run_evals.py score evals/results/scores.jsonl ``` Unit tests in [`tests/test_run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/tests/test_run_evals.py) verify the validation logic, scoring aggregation, release-gate behavior, and budget handling, ensuring the harness itself maintains integrity across evaluation runs. ## Summary - The evaluation pipeline in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) validates cases, executes controlled A/B tests, and scores responses against a weighted rubric. - Skill injection occurs via the `_condition_prompt` function, wrapping [`skills/i-have-adhd/SKILL.md`](https://github.com/ayghri/i-have-adhd/blob/main/skills/i-have-adhd/SKILL.md) in `` tags only for candidate runs. - Five metrics—correctness, autonomy, actionability, safety, and concision—determine release readiness, with safety and correctness acting as blocking criteria. - The release gate passes only when the candidate skill beats baseline scores without regressing on critical dimensions. ## Frequently Asked Questions ### How does the evaluation ensure the skill actually improves responses? The harness generates a run matrix comparing **baseline** (no skill) and **candidate** (with skill) conditions across identical cases. Human judges score both conditions using the same rubric, and `summarize_scores` applies weighted aggregation to quantify improvement. The release gate requires the candidate to exceed baseline scores while maintaining minimum thresholds on safety and correctness. ### What happens if the skill fails the release gate? The evaluation outputs a JSON summary with `passed: false` and explicit failure reasons, such as blocking safety findings or correctness regressions. These results guide targeted modifications to [`skills/i-have-adhd/SKILL.md`](https://github.com/ayghri/i-have-adhd/blob/main/skills/i-have-adhd/SKILL.md) before re-running the evaluation pipeline. ### Can the evaluation harness test skills other than i-have-adhd? Yes. The `--condition-skill` parameter accepts any skill file path. By substituting [`skills/i-have-adhd/SKILL.md`](https://github.com/ayghri/i-have-adhd/blob/main/skills/i-have-adhd/SKILL.md) with an alternative skill definition, the same validation, execution, and scoring infrastructure can evaluate arbitrary response-style modifications across the case catalog. ### Where are the evaluation results stored? Raw LLM responses and metadata write to JSON-L files specified by the `--output` flag (e.g., `evals/results/candidate.jsonl`). Human-judged scores are stored separately, typically in `evals/results/scores.jsonl`, which the `score` sub-command processes to generate the final release-gate summary.