Testing Methodologies in the i-have-adhd Project: A Complete Guide

The i-have-adhd project uses Python's built-in unittest framework for unit testing, temporary-file isolation for safe I/O testing, and GitHub Actions CI workflows for integration validation.

The i-have-adhd repository is a Claude Code plugin that evaluates AI assistant responses against structured test cases. Its testing methodology balances granular unit verification with real-world integration checks to ensure both core logic correctness and runtime compatibility. This article examines the testing architecture, key test files, and practical patterns you can adapt for similar plugin projects.

Unit Testing with Python's unittest Framework

The foundation of the project's testing strategy is tests/test_run_evals.py, which exercises the evaluation harness using Python's standard unittest module. The suite validates every public function in scripts/run_evals.py, including:

• Case validation — ensuring JSON-L records contain required fields
• Score summarization — computing weighted averages across evaluation dimensions
• Duplicate detection — rejecting repeated case IDs
• Release-gate logic — enforcing correctness and safety thresholds
• Budget handling — tracking token or cost limits per run

The test class structure follows standard unittest patterns, with each test method targeting a specific behavior or error condition.

# From tests/test_run_evals.py — example test structure

import unittest
from scripts.run_evals import validate_cases, summarize_scores

class TestEvalHarness(unittest.TestCase):
    def test_validate_cases_rejects_duplicates(self):
        cases = [
            {"id": "case-1", "prompt": "test", "expected": "output"},
            {"id": "case-1", "prompt": "test2", "expected": "output2"},  # duplicate

        ]
        with self.assertRaises(ValueError) as ctx:
            validate_cases(cases)
        self.assertIn("duplicate", str(ctx.exception).lower())

Run the full suite from the repository root:

python -m unittest discover -s tests

Temporary-File Isolation for Safe I/O Testing

File system operations are tested using tempfile.TemporaryDirectory to prevent side effects on the repository. This pattern appears throughout tests/test_run_evals.py (lines 19–31) for simulating malformed inputs and runner configurations.

import json
import tempfile
from pathlib import Path
import unittest
from scripts.run_evals import read_jsonl

class JsonlParsingTest(unittest.TestCase):
    def test_malformed_line_raises_clear_error(self):
        with tempfile.TemporaryDirectory() as td:
            path = Path(td) / "corrupt.jsonl"
            path.write_text(
                json.dumps({"id": "valid-case", "prompt": "test"}) + "\n"
                "this is not valid json\n"
            )
            
            with self.assertRaisesRegex(ValueError, "line 2"):
                read_jsonl(path)

This approach guarantees that:

• Tests remain hermetic and parallelizable
• No manual cleanup is required
• Error messages include precise line numbers for debugging

Edge-Case and Error Condition Coverage

The test suite deliberately injects failure modes that production code must handle gracefully. Key scenarios verified include:

Error Condition	Test Location	Validation Target
Duplicate case IDs	lines 65–90	validate_cases() raises ValueError
Missing required fields	lines 101–119	Schema validation catches omissions
Invalid JSON lines	lines 19–23, 24–31	read_jsonl() provides line-specific errors
Unmetered runner rejection	lines 124–131	Budget-aware runners only
Mismatched condition-row pairings	lines 148–164	Baseline/candidate symmetry enforced

Each assertion verifies both that an exception is raised and that the message contains actionable details for developers.

CI Integration Testing with GitHub Actions

Beyond unit tests, the project validates end-to-end plugin functionality through two GitHub Actions workflows:

Plugin Load Check (.github/workflows/plugin-load-check.yml)

This job installs Claude Code in a fresh environment, registers the plugin, and confirms it reaches "enabled" status. It catches schema-validation errors, missing dependencies, or API incompatibilities that unit tests cannot detect.

# Simplified structure of plugin-load-check.yml

jobs:
  verify-plugin-loads:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install Claude Code
        run: # ... installation steps

      - name: Register and verify plugin
        run: |
          claude plugin install .
          claude plugin status i-have-adhd | grep -q "enabled"

Claude Trigger Workflow (.github/workflows/claude.yml)

Provides additional integration coverage by triggering Claude Code actions on issue comments, ensuring the plugin responds correctly to real interaction patterns.

Together, these workflows provide continuous validation that the plugin remains compatible with the evolving Claude runtime environment.

Manual Evaluation Scaffolding

The repository includes scripts/run_evals.py as a runnable evaluation harness. While not an automated test, it serves as a developer sanity-check for full pipeline execution:

# Manual evaluation run

python scripts/run_evals.py \
  --cases evals/cases.jsonl \
  --runner claude-3-opus \
  --output results/

CI jobs can invoke this script for smoke testing, and developers use it to debug evaluation logic before committing changes.

Extending the Test Suite: Practical Patterns

Adding a Custom Metric Test

When introducing new evaluation dimensions, extend both the WEIGHTS dictionary and the test assertions:

import unittest
from scripts.run_evals import summarize_scores, WEIGHTS

class WeightedMetricTest(unittest.TestCase):
    def test_clarity_metric_integration(self):
        # Register new metric weight

        WEIGHTS["clarity"] = 0.05
        
        scores = [
            {
                "case_id": "test", "trial": 1, "condition": "baseline",
                "correctness": 4, "autonomy": 4, "actionability": 4,
                "safety": 4, "concision": 4, "clarity": 5,
                "blocker": False, "notes": "baseline run"
            },
            {
                "case_id": "test", "trial": 1, "condition": "candidate",
                "correctness": 5, "autonomy": 5, "actionability": 5,
                "safety": 5, "concision": 5, "clarity": 5,
                "blocker": False, "notes": "candidate run"
            },
        ]
        
        summary = summarize_scores(scores)
        self.assertIn("clarity", summary["conditions"]["baseline"])
        self.assertAlmostEqual(
            summary["conditions"]["baseline"]["clarity"], 
            5.0
        )
        
        # Cleanup to prevent cross-test pollution

        del WEIGHTS["clarity"]

Place this in tests/ and run python -m unittest discover -s tests to verify integration.

Key Testing Files Reference

File	Purpose
tests/test_run_evals.py	Core unit-test suite with 15+ test methods covering validation, scoring, and error handling
scripts/run_evals.py	Evaluation engine with functions read_jsonl(), validate_cases(), summarize_scores(), run_evals()
.github/workflows/plugin-load-check.yml	CI validation that plugin loads and enables in Claude Code
.github/workflows/claude.yml	Integration workflow for comment-triggered Claude interactions
evals/README.md	Documentation for evaluation data format, essential for writing valid test fixtures

Summary

• Unit testing uses Python's unittest framework in tests/test_run_evals.py to verify individual functions and error pathways
• Temporary-file isolation ensures hermetic, parallel-safe I/O testing without repository pollution
• Edge-case coverage deliberately injects malformed inputs, duplicates, and schema violations to validate error handling
• CI integration tests via GitHub Actions confirm the plugin loads correctly in fresh Claude Code environments
• Manual scaffolding in scripts/run_evals.py supports exploratory testing and debugging

Frequently Asked Questions

What testing framework does i-have-adhd use?

The project uses Python's built-in unittest framework exclusively. No external test dependencies like pytest are required, keeping the repository lightweight and compatible with standard Python installations.

How does the project test file I/O without side effects?

Tests use tempfile.TemporaryDirectory as a context manager to create isolated file system environments. This pattern appears in tests/test_run_evals.py (lines 19–31) for simulating malformed JSON-L files and runner configurations, ensuring no test artifacts persist after execution.

What integration testing does i-have-adhd perform?

Two GitHub Actions workflows provide integration coverage: plugin-load-check.yml verifies the plugin installs and enables in a fresh Claude Code environment, while claude.yml tests comment-triggered interactions. These catch runtime incompatibilities that unit tests cannot detect.

How can I add a new test for custom evaluation logic?

Create a new unittest.TestCase subclass in tests/, import functions from scripts/run_evals, and follow the existing patterns for temporary-file isolation and assertion styles. Run python -m unittest discover -s tests to validate your addition before submitting changes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}