# What Is the Purpose of the Evaluation Scripts in i-have-adhd?

> Discover the purpose of evaluation scripts in i-have-adhd. Learn how they ensure repeatable, cost-aware validation of AI response quality and safety improvements.

- Repository: [Ayoub Ghriss/i-have-adhd](https://github.com/ayghri/i-have-adhd)
- Tags: how-to-guide
- Published: 2026-08-03

---

**The evaluation scripts in *i-have-adhd* provide a self-contained harness for measuring how well an AI-driven response-style skill performs against a baseline implementation, enabling repeatable, cost-aware validation of safety and quality improvements.**

The *i-have-adhd* repository by `ayghri` contains a disciplined evaluation framework for testing AI-augmented response skills. Understanding what the evaluation scripts do is essential for anyone contributing to or deploying response-style modifications. This article breaks down the four core commands—`validate`, `plan`, `run`, and `score`—as implemented in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py), with specific attention to their purposes in the evaluation workflow.

## Why Evaluation Scripts Matter for Response-Style Skills

Response-style skills modify how an AI assistant communicates with users. Without rigorous evaluation, changes can introduce regressions in safety, correctness, or helpfulness. The *i-have-adhd* evaluation scripts solve this by establishing a **paired comparison methodology**: every test case runs against both a baseline (original behavior) and a candidate (skill-enhanced) condition, with identical prompts and judging criteria.

The framework emphasizes **cost awareness** and **safety gating**. High-risk cases are flagged in `evals/cases.jsonl`, budgets are enforced in dollars per run, and release decisions depend on weighted score aggregates rather than subjective impressions.

## The Four Core Evaluation Commands

### Validate: Ensure Test Case Integrity

Before any execution, the evaluation scripts verify that the test catalog is well-formed. The `validate` sub-command in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) (lines 59-75) checks that every case contains required fields—`id`, `category`, `prompt`, `risk`, and `criteria`—and enforces uniqueness constraints on identifiers.

```bash
python3 scripts/run_evals.py validate

```

This catches schema errors early, preventing wasted computation on malformed test suites. The validation also confirms that risk levels and criteria types match expected enumerations defined in the evaluation schema.

### Plan: Preview the Run Matrix

Reproducibility requires knowing exactly what will execute. The `plan` sub-command (lines 55-62) outputs a JSONL matrix of all `(case, trial, condition)` combinations without invoking any LLM calls.

```bash
python3 scripts/run_evals.py plan --trials 3 --include-comparator

```

The `--include-comparator` flag adds a third condition when comparing against an external baseline. This transparency lets reviewers audit coverage before committing budget, and the JSONL output can be cached or diffed across configuration changes.

### Run: Execute Baseline and Candidate Evaluations

The `run` sub-command (lines 9-34) is the workhorse of the *i-have-adhd* evaluation scripts. It accepts a runner configuration (e.g., Claude), condition label, and optional skill file injection, then generates responses for every cell in the run matrix.

**Baseline execution** (no skill modification):

```bash
python3 scripts/run_evals.py run \
  --runner claude \
  --condition baseline \
  --trials 3 \
  --budget-usd 12.5 \
  --output evals/results/responses.jsonl

```

**Candidate execution** (with skill injection):

```bash
python3 scripts/run_evals.py run \
  --runner claude \
  --condition candidate \
  --condition-skill skills/i-have-adhd/SKILL.md \
  --trials 3 \
  --budget-usd 12.5 \
  --output evals/results/responses.jsonl

```

Key capabilities include:

- **Budget enforcement**: Hard stops when projected costs exceed `--budget-usd`
- **Retry logic with backoff**: Handles transient LLM failures
- **Cost tracking**: Reports actual dollar spend per run
- **JSONL persistence**: Each response appended as a structured line for downstream scoring

The separation of baseline and candidate into distinct command invocations ensures clean experimental conditions—no state leakage between skill versions.

### Score: Aggregate and Gate Releases

After manual judgment populates `scores.jsonl`, the `score` sub-command (lines 30-68) computes weighted aggregates and applies release gates. According to [`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md), the weights are:

| Metric | Weight |
|--------|--------|
| Correctness | 35% |
| Autonomy | 25% |
| Actionability | 20% |
| Safety | 10% |
| Concision | 10% |

```bash
python3 scripts/run_evals.py score evals/results/scores.jsonl

```

The scoring logic checks for **regressions** (candidate underperforms baseline on any metric) and **blockers** (any safety criterion failed). A candidate passes release only if it improves or maintains baseline performance without triggering blocking conditions.

## Key Configuration and Data Files

Understanding the evaluation scripts requires familiarity with these supporting files:

- **[`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py)** – Core CLI with `validate`, `plan`, `run`, and `score` implementations
- **`evals/cases.jsonl`** – Test catalog where each line contains `id`, `category`, `prompt`, `risk`, and `criteria` array
- **[`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md)** – Defines scoring weights and criterion descriptions
- **[`evals/runners.example.json`](https://github.com/ayghri/i-have-adhd/blob/main/evals/runners.example.json)** – Template for configuring LLM runners (API keys, models, rate limits)
- **[`evals/README.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/README.md)** – User-facing workflow documentation

The `cases.jsonl` format enables version-controlled, line-oriented test management. New scenarios are added by appending JSON objects, and `git diff` tracks changes to the evaluation suite over time.

## Safety and Cost Design Patterns

The *i-have-adhd* evaluation scripts embed several defensive patterns relevant to production AI systems:

- **Risk stratification**: Cases tagged with `risk: high` receive additional scrutiny during scoring
- **Triplicate trials**: `--trials 3` provides variance estimates for stochastic LLM outputs
- **Explicit budgeting**: Dollar-denominated limits make tradeoffs tangible
- **Deterministic seeding**: Run matrices are generated consistently for reproducible comparisons

These patterns reflect lessons from evaluating AI systems where output quality varies with temperature, context length, and prompt phrasing.

## Summary

The evaluation scripts in *i-have-adhd* serve four interconnected purposes:

- **Validate** test case integrity before execution
- **Plan** reproducible run matrices for audit and review
- **Run** paired baseline/candidate evaluations with cost controls
- **Score** aggregated results against weighted rubrics with regression and blocker gates

Together, they enable disciplined iteration on response-style skills without compromising safety or breaking existing functionality. The framework is self-contained in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) with configuration in `evals/` and skill definitions in `skills/`.

## Frequently Asked Questions

### What file contains the main evaluation logic?

The core implementation lives in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py), which exports four sub-commands: `validate`, `plan`, `run`, and `score`. Each corresponds to a distinct phase of the evaluation workflow, with source lines for the `run` logic at 9-34 and scoring at 30-68.

### How are test cases structured in i-have-adhd?

Test cases are stored in `evals/cases.jsonl` as newline-delimited JSON objects. Each case requires `id`, `category`, `prompt`, `risk`, and `criteria` fields. The `validate` command enforces this schema and checks identifier uniqueness before any LLM calls execute.

### Can I run evaluations without spending money on API calls?

Yes. The `plan` sub-command generates the full run matrix without invoking any LLM. Use `python3 scripts/run_evals.py plan --trials 3` to preview what would execute, including case coverage and condition pairings, at zero cost.

### What determines whether a candidate skill passes release?

The `score` sub-command applies a weighted rubric (correctness 35%, autonomy 25%, actionability 20%, safety 10%, concision 10%) and checks two blocking conditions: metric regressions against baseline and any failed safety criteria. Both must be clear for release approval.