# Evaluation Tools for the i-have-adhd Repository: A Complete Guide to the Custom Python Harness

> Explore the custom Python evaluation harness for the i-have-adhd repository. Learn about validation, planning, execution, and scoring tools for comprehensive project assessment.

- Repository: [Ayoub Ghriss/i-have-adhd](https://github.com/ayghri/i-have-adhd)
- Tags: how-to-guide
- Published: 2026-08-03

---

**The i-have-adhd project uses a lightweight, self-contained Python evaluation harness centered around [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) with CLI sub-commands for validation, planning, execution, and scoring against a five-dimension rubric.**

The evaluation system in i-have-adhd is purpose-built for comparing LLM responses across controlled conditions. Rather than relying on external benchmarking frameworks, the repository ships with a custom orchestration tool that handles everything from case catalog validation to budget-aware execution and weighted scoring. This article breaks down each component, its source location, and how they work together.

## Core Evaluation Components

The harness consists of four interconnected files that define the complete evaluation pipeline.

### run_evals.py: The Central CLI Driver

Located at [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py), this 210+ line Python module implements the entire evaluation engine. It uses **`argparse`** (lines 6-10) to expose four primary sub-commands:

- **`validate`** – Checks `cases.jsonl` for malformed entries, missing fields, or duplicate `case_id` values
- **`plan`** – Generates a run matrix showing all case × trial × condition combinations as JSONL
- **`run`** – Executes prompts against external LLM runners with USD budget enforcement
- **`score`** – Computes weighted scores against the rubric and determines release-gate status

The driver also imports **`subprocess`** for launching external runners, **`json`** for serialization, and **`shlex`** for safe command formatting in error messages.

### cases.jsonl: The Evaluation Catalog

The test corpus lives at `evals/cases.jsonl`. Each line is a JSON object containing:

- `id` – Unique case identifier
- `category` – Thematic grouping for analysis
- `prompt` – The actual text sent to the LLM
- `risk_level` – Safety classification
- `scoring_criteria` – Per-case rubric overrides (optional)

The `validate` command specifically checks that every required field is present and that no `id` collisions exist.

### rubric.md: Five-Dimension Scoring Framework

Human evaluators use [`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md) to judge responses. The rubric defines five weighted dimensions:

| Dimension | Description |
|-----------|-------------|
| **Correctness** | Factual accuracy of medical/therapeutic claims |
| **Autonomy** | Preservation of user agency in recommendations |
| **Actionability** | Concrete, implementable guidance provided |
| **Safety** | Appropriate handling of risk-level content |
| **Concision** | Brevity without sacrificing completeness |

The rubric also specifies **release-gate conditions**—minimum thresholds that candidate conditions must meet for deployment approval.

### runners.example.json: LLM Provider Configuration

External model invocation is configured via [`evals/runners.example.json`](https://github.com/ayghri/i-have-adhd/blob/main/evals/runners.example.json). The example defines two runners:

1. **Claude** – Includes cost reporting via `--max-budget-usd` flag
2. **Codex** – Outputs structured JSON-L for programmatic consumption

Each runner specification includes the command template, expected response format, and cost extraction rules.

## Budget and Cost Enforcement

The evaluation tools for i-have-adhd include rigorous spend controls. The `run` sub-command accepts `--budget-usd` to cap per-condition expenditure (parsed at lines 35-52 and enforced at lines 84-92).

**Cost reporting requirements:**

- Runners that expose dollar usage (like Claude) work transparently
- Runners without cost data require the `--allow-unmetered` flag
- Exceeding budget triggers immediate termination with partial results preserved

This design enables large-scale evaluations without surprise cloud bills.

## Condition-Based Comparison Design

The harness supports three experimental conditions for A/B-style testing:

- **`baseline`** – Raw LLM without system modifications
- **`candidate`** – LLM with [`skills/i-have-adhd/SKILL.md`](https://github.com/ayghri/i-have-adhd/blob/main/skills/i-have-adhd/SKILL.md) injected into context
- **`comparator`** – Optional third variant for additional baselines (enabled via `--include-comparator`)

Each condition runs for N trials (default: 1) with identical prompts, enabling statistical comparison of the skill file's impact.

## Practical Usage Examples

### Validate the case catalog before running

```bash
python3 scripts/run_evals.py validate

```

### Generate a run plan with 3 trials

```bash
python3 scripts/run_evals.py plan --trials 3 --include-comparator

```

### Execute baseline condition with Claude

```bash
python3 scripts/run_evals.py run \
  --runner claude \
  --condition baseline \
  --trials 3 \
  --budget-usd 12.5 \
  --output evals/results/responses.jsonl

```

### Execute candidate condition with skill injection

```bash
python3 scripts/run_evals.py run \
  --runner claude \
  --condition candidate \
  --condition-skill skills/i-have-adhd/SKILL.md \
  --trials 3 \
  --budget-usd 12.5 \
  --output evals/results/responses.jsonl

```

### Score collected responses

```bash
python3 scripts/run_evals.py score evals/results/responses.jsonl

```

## Output Format and Result Structure

Each execution row in the output JSONL (constructed at lines 90-99) contains:

```json
{
  "case_id": "adhd-focus-001",
  "trial": 2,
  "condition": "candidate",
  "runner": "claude",
  "response": "...",
  "usage": {"input_tokens": 1240, "output_tokens": 890},
  "cost_usd": 0.0034
}

```

The `score` command reads manually-annotated versions of these rows (with dimension scores added), validates required fields, computes per-dimension averages, applies the `WEIGHTS` matrix defined in the scoring logic (lines 30-68, 119-168), and outputs:

- Weighted aggregate scores per condition
- Dimension-by-dimension breakdowns
- **Release-gate pass/fail determination**

## Summary

- **[`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py)** implements a complete evaluation CLI with `validate`, `plan`, `run`, and `score` sub-commands
- **`evals/cases.jsonl`** stores the test case catalog with prompts and metadata
- **[`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md)** defines five weighted scoring dimensions and release criteria
- **[`evals/runners.example.json`](https://github.com/ayghri/i-have-adhd/blob/main/evals/runners.example.json)** configures external LLM invocation (Claude, Codex)
- Budget enforcement via `--budget-usd` and cost reporting prevents runaway spending
- Three conditions (`baseline`, `candidate`, `comparator`) enable controlled A/B testing of the skill file
- All components are self-contained—no external orchestration platform required

## Frequently Asked Questions

### Does i-have-adhd use an external evaluation framework like EleutherAI lm-eval?

No. According to the ayghri/i-have-adhd source code, the project uses a custom Python harness in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) rather than third-party frameworks. This keeps dependencies minimal and allows tight integration with the repository's specific case structure and rubric.

### How does the scoring system handle subjectivity in human judgments?

The rubric in [`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md) operationalizes each dimension with concrete criteria. The `score` command then averages multiple trial scores and applies fixed weights—reducing individual rater variance through aggregation and standardization.

### Can I evaluate with my own LLM provider?

Yes. The [`runners.example.json`](https://github.com/ayghri/i-have-adhd/blob/main/runners.example.json) file demonstrates the required configuration format. You can define new runners by specifying the command template, response parsing rules, and (optionally) cost extraction pattern. Use `--allow-unmetered` if your runner doesn't report dollar costs.

### What happens if my evaluation exceeds the budget?

The `run` command tracks cumulative spend against `--budget-usd`. If a response would exceed the cap, the harness terminates immediately and writes partial results to the specified output file. This protects against unexpected charges during large batch runs.