What Is the Purpose of the Evaluation Scripts in i-have-adhd?

The evaluation scripts in i-have-adhd provide a self-contained harness for measuring how well an AI-driven response-style skill performs against a baseline implementation, enabling repeatable, cost-aware validation of safety and quality improvements.

The i-have-adhd repository by ayghri contains a disciplined evaluation framework for testing AI-augmented response skills. Understanding what the evaluation scripts do is essential for anyone contributing to or deploying response-style modifications. This article breaks down the four core commands—validate, plan, run, and score—as implemented in scripts/run_evals.py, with specific attention to their purposes in the evaluation workflow.

Why Evaluation Scripts Matter for Response-Style Skills

Response-style skills modify how an AI assistant communicates with users. Without rigorous evaluation, changes can introduce regressions in safety, correctness, or helpfulness. The i-have-adhd evaluation scripts solve this by establishing a paired comparison methodology: every test case runs against both a baseline (original behavior) and a candidate (skill-enhanced) condition, with identical prompts and judging criteria.

The framework emphasizes cost awareness and safety gating. High-risk cases are flagged in evals/cases.jsonl, budgets are enforced in dollars per run, and release decisions depend on weighted score aggregates rather than subjective impressions.

The Four Core Evaluation Commands

Validate: Ensure Test Case Integrity

Before any execution, the evaluation scripts verify that the test catalog is well-formed. The validate sub-command in scripts/run_evals.py (lines 59-75) checks that every case contains required fields—id, category, prompt, risk, and criteria—and enforces uniqueness constraints on identifiers.

python3 scripts/run_evals.py validate

This catches schema errors early, preventing wasted computation on malformed test suites. The validation also confirms that risk levels and criteria types match expected enumerations defined in the evaluation schema.

Plan: Preview the Run Matrix

Reproducibility requires knowing exactly what will execute. The plan sub-command (lines 55-62) outputs a JSONL matrix of all (case, trial, condition) combinations without invoking any LLM calls.

python3 scripts/run_evals.py plan --trials 3 --include-comparator

The --include-comparator flag adds a third condition when comparing against an external baseline. This transparency lets reviewers audit coverage before committing budget, and the JSONL output can be cached or diffed across configuration changes.

Run: Execute Baseline and Candidate Evaluations

The run sub-command (lines 9-34) is the workhorse of the i-have-adhd evaluation scripts. It accepts a runner configuration (e.g., Claude), condition label, and optional skill file injection, then generates responses for every cell in the run matrix.

Baseline execution (no skill modification):

python3 scripts/run_evals.py run \
  --runner claude \
  --condition baseline \
  --trials 3 \
  --budget-usd 12.5 \
  --output evals/results/responses.jsonl

Candidate execution (with skill injection):

python3 scripts/run_evals.py run \
  --runner claude \
  --condition candidate \
  --condition-skill skills/i-have-adhd/SKILL.md \
  --trials 3 \
  --budget-usd 12.5 \
  --output evals/results/responses.jsonl

Key capabilities include:

• Budget enforcement: Hard stops when projected costs exceed --budget-usd
• Retry logic with backoff: Handles transient LLM failures
• Cost tracking: Reports actual dollar spend per run
• JSONL persistence: Each response appended as a structured line for downstream scoring

The separation of baseline and candidate into distinct command invocations ensures clean experimental conditions—no state leakage between skill versions.

Score: Aggregate and Gate Releases

After manual judgment populates scores.jsonl, the score sub-command (lines 30-68) computes weighted aggregates and applies release gates. According to evals/rubric.md, the weights are:

Metric	Weight
Correctness	35%
Autonomy	25%
Actionability	20%
Safety	10%
Concision	10%

python3 scripts/run_evals.py score evals/results/scores.jsonl

The scoring logic checks for regressions (candidate underperforms baseline on any metric) and blockers (any safety criterion failed). A candidate passes release only if it improves or maintains baseline performance without triggering blocking conditions.

Key Configuration and Data Files

Understanding the evaluation scripts requires familiarity with these supporting files:

• scripts/run_evals.py – Core CLI with validate, plan, run, and score implementations
• evals/cases.jsonl – Test catalog where each line contains id, category, prompt, risk, and criteria array
• evals/rubric.md – Defines scoring weights and criterion descriptions
• evals/runners.example.json – Template for configuring LLM runners (API keys, models, rate limits)
• evals/README.md – User-facing workflow documentation

The cases.jsonl format enables version-controlled, line-oriented test management. New scenarios are added by appending JSON objects, and git diff tracks changes to the evaluation suite over time.

Safety and Cost Design Patterns

The i-have-adhd evaluation scripts embed several defensive patterns relevant to production AI systems:

• Risk stratification: Cases tagged with risk: high receive additional scrutiny during scoring
• Triplicate trials: --trials 3 provides variance estimates for stochastic LLM outputs
• Explicit budgeting: Dollar-denominated limits make tradeoffs tangible
• Deterministic seeding: Run matrices are generated consistently for reproducible comparisons

These patterns reflect lessons from evaluating AI systems where output quality varies with temperature, context length, and prompt phrasing.

Summary

The evaluation scripts in i-have-adhd serve four interconnected purposes:

• Validate test case integrity before execution
• Plan reproducible run matrices for audit and review
• Run paired baseline/candidate evaluations with cost controls
• Score aggregated results against weighted rubrics with regression and blocker gates

Together, they enable disciplined iteration on response-style skills without compromising safety or breaking existing functionality. The framework is self-contained in scripts/run_evals.py with configuration in evals/ and skill definitions in skills/.

Frequently Asked Questions

What file contains the main evaluation logic?

The core implementation lives in scripts/run_evals.py, which exports four sub-commands: validate, plan, run, and score. Each corresponds to a distinct phase of the evaluation workflow, with source lines for the run logic at 9-34 and scoring at 30-68.

How are test cases structured in i-have-adhd?

Test cases are stored in evals/cases.jsonl as newline-delimited JSON objects. Each case requires id, category, prompt, risk, and criteria fields. The validate command enforces this schema and checks identifier uniqueness before any LLM calls execute.

Can I run evaluations without spending money on API calls?

Yes. The plan sub-command generates the full run matrix without invoking any LLM. Use python3 scripts/run_evals.py plan --trials 3 to preview what would execute, including case coverage and condition pairings, at zero cost.

What determines whether a candidate skill passes release?

The score sub-command applies a weighted rubric (correctness 35%, autonomy 25%, actionability 20%, safety 10%, concision 10%) and checks two blocking conditions: metric regressions against baseline and any failed safety criteria. Both must be clear for release approval.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}