How Evaluations Reflect Real‑World Use Cases for i‑have‑adhd: A Technical Breakdown

The i‑have‑adhd evaluation harness mirrors production developer workflows by running paired baseline versus skill‑enhanced trials against realistic task prompts, measuring outputs against the skill's own quality rubric, and enforcing release gates that match production deployment criteria.

The i‑have‑adhd repository provides an end‑to‑end evaluation system that validates whether its ADHD‑optimized response style actually improves real developer interactions. The evaluation design directly simulates how users would encounter this skill in production—same prompts, same cost constraints, same quality expectations.

Real‑World Prompt Simulation Through evals/cases.jsonl

The foundation of realistic evaluation lies in the case catalog at evals/cases.jsonl. This JSON‑Lines file contains concrete, task‑oriented prompts that mirror genuine developer queries.

Each case represents something an ADHD reader might actually ask:

• Adding a new route to a Node.js application
• Explaining OAuth flow implementation
• Debugging asynchronous code patterns

The format captures not just the prompt text but metadata that preserves context for reproducible trials. Inspect the structure directly:

from pathlib import Path
import json

cases = [json.loads(line) for line in Path("evals/cases.jsonl").read_text().splitlines()]
print(json.dumps(cases[0], indent=2))

This approach ensures every evaluation run exercises the skill against authentic interaction scenarios rather than synthetic or simplified test inputs.

Paired Execution: Reproducing User Sessions with scripts/run_evals.py

The evaluation driver in scripts/run_evals.py implements a paired‑condition design that reproduces how a user would experience the skill in practice. It executes identical prompts through both baseline (raw model) and candidate (skill‑enhanced) pipelines.

Run a complete paired evaluation cycle:

# Validate case catalog integrity

python3 scripts/run_evals.py validate

# Execute baseline condition — raw model output

python3 scripts/run_evals.py run \
  --runner claude \
  --condition baseline \
  --trials 3 \
  --budget-usd 12.50 \
  --output evals/results/responses.jsonl

# Execute candidate condition — with ADHD skill applied

python3 scripts/run_evals.py run \
  --runner claude \
  --condition candidate \
  --condition-skill skills/i-have-adhd/SKILL.md \
  --trials 3 \
  --budget-usd 12.50 \
  --output evals/results/responses.jsonl

Key real‑world fidelity features in this implementation:

• Budget enforcement: The --budget-usd parameter mirrors production cost constraints
• Configuration isolation: --setting-sources "" prevents external plugins or memory from contaminating results—simulating a clean deployment environment
• Deduplication logic: The harness tracks completed rows, allowing incremental evaluation runs without redundant API calls
• Trial multiplicity: Multiple trials per case capture response variance, reflecting the non‑deterministic nature of live usage

Quality Measurement Aligned with Skill Promises in evals/rubric.md

The rubric at evals/rubric.md translates the skill's design goals into measurable evaluation dimensions. This directly connects evaluation criteria to real‑world user needs.

The rubric defines five weighted dimensions:

Dimension	Weight	Real‑World Priority
Correctness	30%	Output must be factually accurate—users with ADHD cannot afford to untangle errors
Autonomy	25%	Reader can act without follow‑up questions ("lead with the next action")
Actionability	20%	Concrete steps, not abstract explanations
Safety	15%	No harmful or misleading guidance
Concision	10%	Eliminating filler that taxes working memory

These weights encode the skill's core promise from skills/i-have-adhd/SKILL.md: responses that respect cognitive load constraints while preserving utility. Blind scoring ensures objective comparison between conditions.

Release Gates Matching Production Standards

The summarize_scores function (invoked via the score subcommand) implements automated release criteria that parallel production deployment checks:

# After judges produce scores.jsonl, aggregate and gate

python3 scripts/run_evals.py score evals/results/scores.jsonl

The scoring logic enforces three hard requirements derived from the skill's operational rules:

1. No blocking findings — any safety or correctness failure blocks release
2. Baseline parity — candidate must match or exceed baseline correctness and safety scores
3. Weighted improvement — candidate's overall weighted score must exceed baseline

This mirrors how organizations actually ship model‑enhancement features: quantitative thresholds that protect user experience while enabling iteration.

Isolation and Reproducibility as Production Mirrors

The harness enforces environmental constraints that replicate controlled production deployment:

• Deterministic case ordering ensures comparable runs across evaluation cycles
• Budget caps prevent runaway costs during testing
• Skill‑only configuration eliminates confounding variables from other extensions
• Structured output paths enable automated result comparison and archiving

These mechanisms make the evaluation itself a reproducible artifact that can be audited, versioned, and integrated into CI pipelines—just like the skill it validates.

Summary

• Realistic prompts: evals/cases.jsonl contains authentic developer tasks, not synthetic test inputs
• Paired execution: scripts/run_evals.py runs identical prompts through both raw and skill‑enhanced models to isolate impact
• Aligned rubric: evals/rubric.md weights reflect the skill's actual design priorities for ADHD readers
• Production gates: Scoring logic enforces release criteria matching operational deployment standards
• Clean environment: Budget controls and configuration isolation simulate controlled production conditions

Frequently Asked Questions

What makes the evaluation prompts realistic rather than artificial?

The case catalog at evals/cases.jsonl draws from concrete developer scenarios—adding API routes, understanding authentication flows, debugging specific patterns. These mirror the task‑focused queries that ADHD readers actually pose when seeking coding assistance, rather than abstract or simplified test questions.

How does the paired design improve evaluation validity?

By feeding identical prompts to both baseline and candidate conditions in scripts/run_evals.py, the harness controls for prompt variability. Any quality difference between outputs can be attributed to the skill itself rather than prompt difficulty or phrasing differences, strengthening causal inference about the skill's real‑world impact.

Why does the rubric weight Autonomy higher than Concision?

At 25% versus 10%, Autonomy reflects that actionable independence matters more than brevity alone for the target audience. A concise but incomplete answer still requires follow‑up work, whereas an autonomous response enables immediate progress—aligning with the skill's core promise of reducing cognitive overhead.

Can organizations adapt this evaluation approach for other response‑style skills?

Yes. The harness structure—case catalogs, paired execution, weighted rubrics, and automated gating—transfers directly. Organizations would modify evals/cases.jsonl for their domain, adjust weights in evals/rubric.md for their quality priorities, and adapt the skill path in scripts/run_evals.py while preserving the core validation workflow.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}