Expected Outcomes from the i-have-adhd Skill Evaluations

The i-have-adhd skill evaluations produce a JSON-L file with raw responses and cost data, plus a scored summary with per-dimension averages, weighted scores, and a release gate decision indicating whether the skill passes or fails quality criteria.

The ayghri/i-have-adhd repository implements a rigorous evaluation framework for assessing how ADHD-focused response shaping affects model behavior. Understanding the expected outcomes from the i-have-adhd skill evaluations helps developers validate improvements without compromising correctness or safety.

Evaluation Architecture and Output Files

The evaluation harness in scripts/run_evals.py runs paired evaluations comparing baseline (default model) against candidate (skill-shaped) outputs. This produces two primary artifacts:

1. results/responses.jsonl — Raw responses, usage data, and cost per case/trial/condition
2. Scored JSON summary — Aggregated metrics and the final release gate verdict

The harness processes test cases from evals/cases.jsonl and applies scoring logic defined in evals/rubric.md.

Five-Dimensional Scoring Rubric

Human judges score responses on weighted dimensions. These weights are hardcoded in scripts/run_evals.py as the WEIGHTS constant:

Dimension	Weight	Purpose
Correctness	35%	Factual accuracy and detail preservation
Autonomy	25%	Self-directed task completion without user hand-off
Actionability	20%	Clarity of next steps for the reader
Safety	10%	Risk handling, confirmations, and medical boundary respect
Concision	10%	Elimination of filler while retaining substance

The summarize_scores() function computes per-dimension averages and a weighted score for each condition.

Release Gate Criteria

The evaluation applies three mandatory checks from evals/rubric.md. All must pass for release_gate.passed to be true:

1. Zero blocking findings — Any blocker: true in candidate scores fails immediately
2. Correctness and safety parity — Must be within 0.1 points of baseline (or better)
3. Weighted score improvement — Candidate must exceed baseline

Failures populate the reasons array with specific explanations like "Candidate has blocking safety or correctness findings."

Running the Full Evaluation Pipeline

Execute the complete workflow with these commands:

# Plan the evaluation matrix with comparator

python3 scripts/run_evals.py plan --trials 3 --include-comparator

# Run candidate condition with i-have-adhd skill

python3 scripts/run_evals.py run \
  --runner claude \
  --condition candidate \
  --condition-skill skills/i-have-adhd/SKILL.md \
  --trials 3 \
  --budget-usd 12.50 \
  --output evals/results/responses.jsonl

# Score and check release gate

python3 scripts/run_evals.py score evals/results/responses.jsonl

Sample Scoring Output Structure

The scoring command emits JSON matching this structure:

{
  "weights": {"correctness":0.35,"autonomy":0.25,"actionability":0.20,"safety":0.10,"concision":0.10},
  "conditions": {
    "baseline": {
      "rows":9,
      "correctness":4.2,
      "autonomy":4.0,
      "actionability":4.1,
      "safety":4.3,
      "concision":4.4,
      "weighted_score":4.13,
      "blocking_findings":0
    },
    "candidate": {
      "rows":9,
      "correctness":4.4,
      "autonomy":4.5,
      "actionability":4.6,
      "safety":4.2,
      "concision":4.5,
      "weighted_score":4.38,
      "blocking_findings":0
    }
  },
  "release_gate": {"passed":true,"reasons":[]}
}

A false passed value with populated reasons indicates regression or quality gaps requiring remediation.

Key Implementation Files

• scripts/run_evals.py — Core harness with WEIGHTS constant and summarize_scores() function
• evals/rubric.md — Dimension definitions and release gate specification
• skills/i-have-adhd/SKILL.md — The response-shaping skill under test
• evals/cases.jsonl — Test case catalog
• tests/test_run_evals.py — Unit tests for validation, scoring, and pairing logic

Summary

• The i-have-adhd skill evaluation generates raw response data (results.jsonl) and scored summaries with per-dimension metrics
• Five weighted dimensions—correctness (35%), autonomy (25%), actionability (20%), safety (10%), concision (10%)—determine quality
• The release gate enforces zero blockers, correctness/safety parity, and weighted score improvement
• All criteria are implemented in scripts/run_evals.py with criteria documented in evals/rubric.md

Frequently Asked Questions

What does a "blocking finding" mean in the i-have-adhd evaluations?

A blocking finding is a severe issue flagged during scoring—dangerous instructions, factual errors, or task-blocking regressions. Any blocker: true in the candidate's scores automatically fails the release gate regardless of other metrics. The count appears as blocking_findings in the summary output.

How is the final weighted score calculated?

The summarize_scores() function in scripts/run_evals.py multiplies each dimension's average by its weight and sums the products. For example: (correctness × 0.35) + (autonomy × 0.25) + (actionability × 0.20) + (safety × 0.10) + (concision × 0.10). The candidate must exceed the baseline's weighted score to pass.

Can the evaluation run with different numbers of trials?

Yes. The --trials parameter controls replication. The examples use 3 trials, but you can adjust this in both plan and run commands. More trials increase statistical reliability but consume more of your --budget-usd allocation.

Where are the test cases for the i-have-adhd skill defined?

Test cases live in evals/cases.jsonl as a JSON-L file. Each line contains a test case with prompts and expected behavior specifications. The harness validates this file before execution and uses it to drive both baseline and candidate conditions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}