How the i-have-adhd Skill Is Evaluated: A Complete Technical Guide

The i-have-adhd skill is evaluated through a three-stage pipeline that validates test cases, executes LLM runs with and without the skill injected, and scores responses against a weighted rubric to determine release readiness.

The ayghri/i-have-adhd repository implements a rigorous evaluation harness to objectively measure whether the ADHD-specific response skill improves LLM outputs without compromising safety or correctness. Understanding how the i-have-adhd skill is evaluated requires examining the orchestration logic in scripts/run_evals.py, the case catalog structure, and the scoring methodology that gates releases.

The Three-Stage Evaluation Pipeline

The evaluation process follows a structured workflow defined in scripts/run_evals.py, moving from case validation through execution to final scoring.

Stage 1: Validation and Planning

Before any LLM calls occur, the harness validates the case catalog stored in evals/cases.jsonl. The validate_cases sub-command checks for required fields, uniqueness constraints, and proper risk categorization across scenarios like "direct-answer", "multi-step-progress", and "medical-boundary".

The plan sub-command then generates a run matrix that pairs every validated case with two conditions: baseline (no skill) and candidate (with the i-have-adhd skill). This matrix ensures systematic A/B testing across the entire scenario catalog.

Stage 2: Execution and Skill Injection

The run_evaluations function executes the planned matrix, launching the selected LLM runner (e.g., Claude) for each case-trial pair. The critical logic resides in _condition_prompt, which handles conditional prompt construction:

• Baseline runs: Receive only the raw task prompt from the case definition.
• Candidate runs: Inject the skill file content into the prompt via a <response_style> wrapper, forcing the model to obey the i-have-adhd response rules while answering the original task.

The skill content is read from skills/i-have-adhd/SKILL.md and wrapped in XML-style tags before the task prompt. Results including response text, token usage, and cost are written to JSON-L files for downstream analysis.

Stage 3: Scoring and Release-Gate Analysis

Human judges score each response using the rubric defined in evals/rubric.md. The summarize_scores function aggregates these scores across five weighted metrics: correctness, autonomy, actionability, safety, and concision.

The release gate applies strict pass/fail logic:

• The candidate must not regress on safety or correctness compared to baseline.
• No blocking findings can be present in the evaluation.
• The candidate must exceed the baseline's weighted aggregate score.

Only when all criteria are met does the harness set passed: true in the final JSON summary.

Core Evaluation Concepts

Case Catalog Structure

The evals/cases.jsonl file contains the test scenarios that exercise the skill across diverse interaction patterns. Each JSON object defines the prompt context, risk level, and evaluation criteria. Cases span from simple direct answers to complex multi-step planning scenarios, ensuring the skill performs robustly across conversation types.

Skill Injection via response_style

The evaluation isolates the skill's impact by strictly controlling when rules apply. During candidate runs, scripts/run_evals.py reads skills/i-have-adhd/SKILL.md and injects it into the prompt structure:

# Conceptual injection pattern used by _condition_prompt

<response_style>
[Content of skills/i-have-adhd/SKILL.md]
</response_style>

[Original task prompt from case]

This injection method ensures the model receives explicit formatting instructions without altering the underlying task difficulty, creating a fair comparison against baseline performance.

The Five-Metric Scoring Rubric

Scores are assigned on a 1-5 scale across five dimensions defined in evals/rubric.md:

• Correctness: Factual accuracy and relevance to the query.
• Autonomy: Appropriate level of independence vs. requesting clarification.
• Actionability: Concrete, implementable guidance provided.
• Safety: Absence of harmful content and proper handling of sensitive topics.
• Concision: Efficiency of communication without excessive verbosity.

The WEIGHTS mapping in scripts/run_evals.py applies these priorities during aggregation, with safety and correctness typically carrying higher weight in the final calculation.

Release Gate Criteria

The release gate prevents regression by enforcing minimum performance thresholds. The harness compares candidate scores against baseline results, checking for statistically significant improvements without trade-offs in critical dimensions. Failure reasons are explicitly logged, such as "Candidate has blocking safety or correctness findings," enabling targeted skill refinement.

Running the Evaluation Harness

Execute the full evaluation pipeline using the CLI commands defined in scripts/run_evals.py:

# Step 1: Validate the case catalog integrity

python3 scripts/run_evals.py validate

# Step 2: Generate the run matrix with baseline and candidate conditions

python3 scripts/run_evals.py plan --trials 3 --include-comparator > run_matrix.jsonl

# Step 3: Execute baseline condition (no skill injection)

python3 scripts/run_evals.py run \
  --runner claude \
  --condition baseline \
  --trials 3 \
  --budget-usd 12.50 \
  --output evals/results/baseline.jsonl

# Step 4: Execute candidate condition with i-have-adhd skill injected

python3 scripts/run_evals.py run \
  --runner claude \
  --condition candidate \
  --condition-skill skills/i-have-adhd/SKILL.md \
  --trials 3 \
  --budget-usd 12.50 \
  --output evals/results/candidate.jsonl

# Step 5: Score results and check release gate

python3 scripts/run_evals.py score evals/results/scores.jsonl

Unit tests in tests/test_run_evals.py verify the validation logic, scoring aggregation, release-gate behavior, and budget handling, ensuring the harness itself maintains integrity across evaluation runs.

Summary

• The evaluation pipeline in scripts/run_evals.py validates cases, executes controlled A/B tests, and scores responses against a weighted rubric.
• Skill injection occurs via the _condition_prompt function, wrapping skills/i-have-adhd/SKILL.md in <response_style> tags only for candidate runs.
• Five metrics—correctness, autonomy, actionability, safety, and concision—determine release readiness, with safety and correctness acting as blocking criteria.
• The release gate passes only when the candidate skill beats baseline scores without regressing on critical dimensions.

Frequently Asked Questions

How does the evaluation ensure the skill actually improves responses?

The harness generates a run matrix comparing baseline (no skill) and candidate (with skill) conditions across identical cases. Human judges score both conditions using the same rubric, and summarize_scores applies weighted aggregation to quantify improvement. The release gate requires the candidate to exceed baseline scores while maintaining minimum thresholds on safety and correctness.

What happens if the skill fails the release gate?

The evaluation outputs a JSON summary with passed: false and explicit failure reasons, such as blocking safety findings or correctness regressions. These results guide targeted modifications to skills/i-have-adhd/SKILL.md before re-running the evaluation pipeline.

Can the evaluation harness test skills other than i-have-adhd?

Yes. The --condition-skill parameter accepts any skill file path. By substituting skills/i-have-adhd/SKILL.md with an alternative skill definition, the same validation, execution, and scoring infrastructure can evaluate arbitrary response-style modifications across the case catalog.

Where are the evaluation results stored?

Raw LLM responses and metadata write to JSON-L files specified by the --output flag (e.g., evals/results/candidate.jsonl). Human-judged scores are stored separately, typically in evals/results/scores.jsonl, which the score sub-command processes to generate the final release-gate summary.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}