Writing Evaluation Tests for the i-have-adhd Skill: A Complete Guide

The i-have-adhd repository provides a lightweight pytest-based test suite in tests/test_run_evals.py to verify that the evaluation harness (scripts/run_evals.py) functions correctly, with separate test functions for each subcommand including validate, plan, and scoring workflows.

The i-have-adhd project is a Claude skill that reformats assistant outputs for ADHD readers. Because the skill itself contains no runtime code—it operates entirely through declarative rules in [SKILL.md](https://github.com/ayghri/i-have-adhd/blob/main/skills/i-have-adhd/SKILL.md)—the primary verification mechanism is the evaluation harness. Writing robust evaluation tests ensures this harness correctly validates skill files, plans trials, and scores outputs against baseline comparisons. This guide explains how to write and extend these evaluation tests based on the repository's existing patterns.

Core Evaluation Test Architecture

The test suite follows a simple, modular structure that mirrors the subcommand architecture of the evaluation script.

Test File Organization

Located at [tests/test_run_evals.py](https://github.com/ayghri/i-have-adhd/blob/main/tests/test_run_evals.py), the test file provides placeholder functions for each major evaluation operation:

import pytest

def test_validate():
    # Placeholder test

    assert True

def test_plan():
    # Placeholder...

    assert ...

Each function corresponds to a subcommand in [scripts/run_evals.py](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py). The CLI driver exposes four primary operations: validate, plan, run, and score—though the test file currently focuses on the core validation and planning workflows.

Subcommand Mapping in the Evaluation Script

The evaluation harness in scripts/run_evals.py implements these subcommands through a dispatch pattern:

"""Evaluates this skill.

Usage: python3 scripts/run_evals.py [SUBCOMMAND] ...

Subcommands:
    validate
        Validate the output of the skill.
    plan
        ...
"""
import argparse
import json
import os
import subprocess
from pathlib import Path

def _run_subcommand(subcommand: str, args: list[str]):
    # ... placeholder...

    pass

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Run evaluation")
    # ...

Writing Tests for Each Evaluation Phase

To write comprehensive evaluation tests, target the three-phase evaluation workflow documented in [evals/README.md](https://github.com/ayghri/i-have-adhd/blob/main/evals/README.md): validation, execution planning, and scoring.

Phase 1: Skill Validation Tests

The validate subcommand checks that skill files are properly formatted and conform to expected schemas. A proper test should verify:

• YAML frontmatter parsing in SKILL.md
• Required fields: name, description, rules
• JSON syntax validity in plugin.json

def test_validate_skill_md_exists():
    """Verify SKILL.md file exists and contains required YAML header."""
    skill_path = Path("skills/i-have-adhd/SKILL.md")
    assert skill_path.exists(), "SKILL.md must exist"
    
    content = skill_path.read_text()
    assert content.startswith("---"), "YAML frontmatter required"
    assert "name: i-have-adhd" in content, "Skill name must be declared"

def test_validate_plugin_json():
    """Verify plugin.json contains required metadata fields."""
    import json
    
    plugin_path = Path("plugin.json")
    assert plugin_path.exists(), "plugin.json must exist"
    
    with open(plugin_path) as f:
        data = json.load(f)  # Validates JSON syntax

    
    assert "name" in data, "Plugin name required"
    assert "description" in data, "Plugin description required"

Phase 2: Execution Planning Tests

The plan subcommand generates trial configurations. Tests should verify that planning correctly:

• Accepts trial count parameters (--trials)
• Handles comparator inclusion flags (--include-comparator)
• Produces valid execution manifests

def test_plan_generates_correct_trial_count():
    """Verify plan subcommand respects --trials parameter."""
    from scripts.run_evals import _run_subcommand
    
    # Mock arguments for 5 trials with comparator

    result = _run_subcommand("plan", ["--trials", "5", "--include-comparator"])
    assert result["baseline_trials"] == 5
    assert result["candidate_trials"] == 5
    assert result["include_comparator"] is True

def test_plan_default_trials():
    """Verify sensible defaults when trials not specified."""
    result = _run_subcommand("plan", [])
    assert result["baseline_trials"] >= 1, "Must generate at least one trial"

Phase 3: Runner Integration Tests

The run subcommand executes evaluations against specified runners (e.g., Claude). Critical test scenarios include:

• Budget enforcement (--budget-usd)
• Condition specification (baseline vs candidate)
• Output file generation

import tempfile
import json

def test_run_outputs_valid_jsonl():
    """Verify run subcommand produces parseable JSONL output."""
    with tempfile.NamedTemporaryFile(mode='w', suffix='.jsonl', delete=False) as tmp:
        _run_subcommand("run", [
            "--runner", "claude",
            "--condition", "baseline",
            "--trials", "2",
            "--output", tmp.name
        ])
        
        # Validate JSONL format

        with open(tmp.name) as f:
            for line in f:
                record = json.loads(line)  # Raises if invalid JSON

                assert "response" in record
                assert "trial_id" in record

Testing the Scoring Pipeline

The final evaluation phase compares baseline and candidate outputs. Tests should verify that scoring:

• Correctly applies the evaluation rubric from [evals/rubric.md](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md)
• Generates comparable metrics (ADHD-style adherence scores)
• Handles missing or malformed responses gracefully

def test_score_applies_adhd_rubric():
    """Verify scoring uses ADHD-specific criteria from rubric.md."""
    from scripts.run_evals import _run_subcommand
    
    test_responses = [
        {"response": "First, run `npm install`. (2 min)", "condition": "candidate"},
        {"response": "I'd be happy to help you with that installation...", "condition": "baseline"}
    ]
    
    scores = _run_subcommand("score", ["--input", "-"], input_data=test_responses)
    
    # Candidate should score higher on "next-action-first" criterion

    assert scores["candidate"]["next_action_score"] > scores["baseline"]["next_action_score"]

Integration Testing Patterns

For end-to-end verification, combine all phases in a single test flow:

def test_full_evaluation_pipeline(tmp_path):
    """Execute complete validation-plan-run-score cycle."""
    from scripts.run_evals import _run_subcommand
    
    # Phase 1: Validate

    assert _run_subcommand("validate", [])["valid"] is True
    
    # Phase 2: Plan

    plan = _run_subcommand("plan", ["--trials", "3"])
    assert plan["total_trials"] == 6  # 3 baseline + 3 candidate

    
    # Phase 3: Run (with mock runner for speed)

    output_file = tmp_path / "responses.jsonl"
    _run_subcommand("run", [
        "--runner", "mock",
        "--trials", "3",
        "--output", str(output_file)
    ])
    assert output_file.exists()
    
    # Phase 4: Score

    score_file = tmp_path / "scores.jsonl"
    _run_subcommand("score", [
        "--input", str(output_file),
        "--output", str(score_file)
    ])
    assert score_file.exists()

Extending the Test Suite

When modifying [scripts/run_evals.py](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py), follow these testing practices:

1. Add matching test functions for new subcommands in tests/test_run_evals.py
2. Parameterize budget and trial configurations to catch edge cases
3. Mock external API calls to the Claude runner to ensure tests run without credentials
4. Validate output schemas against expected JSON structures
5. Include rubric-specific assertions that verify ADHD-style formatting rules are correctly evaluated

Summary

• Test file location: [tests/test_run_evals.py](https://github.com/ayghri/i-have-adhd/blob/main/tests/test_run_evals.py) contains the pytest suite for evaluation verification
• Subcommand coverage: Write separate tests for validate, plan, run, and score operations
• Validation tests: Verify SKILL.md YAML frontmatter and plugin.json schema compliance
• Planning tests: Check trial count handling and comparator flag processing
• Execution tests: Confirm JSONL output generation and runner parameter passing
• Scoring tests: Validate rubric application from [evals/rubric.md](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md)
• Integration pattern: Combine all phases in pipeline tests for end-to-end verification

Frequently Asked Questions

How do I run the existing evaluation tests?

Execute pytest tests/test_run_evals.py from the repository root. The current placeholder tests will pass immediately; replace the assert True and assert ... statements with actual validation logic as you implement the evaluation harness.

What should I test if the evaluation script is still a placeholder?

Focus on interface contracts: verify that _run_subcommand("validate", []) returns a dictionary with a "valid" boolean, that plan accepts integer --trials values, and that run creates output files at the specified paths. These tests prevent regressions as the implementation evolves.

Why does a declarative skill need evaluation tests?

The skill itself has no code, but the evaluation harness that validates it does. Tests ensure this harness correctly parses skill metadata, executes comparative trials against baseline Claude responses, and accurately scores ADHD-style formatting adherence using the evaluation rubric.

How do I test budget enforcement in the run subcommand?

Pass --budget-usd with a very low value (e.g., 0.01) and verify the runner terminates early or raises a budget-exceeded error. Alternatively, mock the cost-tracking function and assert it accumulates charges correctly across trials.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}