# Testing Methodologies in the i-have-adhd Project: A Complete Guide

> Explore the testing methodologies for the i-have-adhd project. Learn about unittest, I/O testing, and GitHub Actions CI for robust validation.

- Repository: [Ayoub Ghriss/i-have-adhd](https://github.com/ayghri/i-have-adhd)
- Tags: how-to-guide
- Published: 2026-08-03

---

**The i-have-adhd project uses Python's built-in `unittest` framework for unit testing, temporary-file isolation for safe I/O testing, and GitHub Actions CI workflows for integration validation.**

The **i-have-adhd** repository is a Claude Code plugin that evaluates AI assistant responses against structured test cases. Its testing methodology balances granular unit verification with real-world integration checks to ensure both core logic correctness and runtime compatibility. This article examines the testing architecture, key test files, and practical patterns you can adapt for similar plugin projects.

## Unit Testing with Python's unittest Framework

The foundation of the project's testing strategy is [`tests/test_run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/tests/test_run_evals.py), which exercises the evaluation harness using Python's standard `unittest` module. The suite validates every public function in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py), including:

- **Case validation** — ensuring JSON-L records contain required fields
- **Score summarization** — computing weighted averages across evaluation dimensions
- **Duplicate detection** — rejecting repeated case IDs
- **Release-gate logic** — enforcing correctness and safety thresholds
- **Budget handling** — tracking token or cost limits per run

The test class structure follows standard `unittest` patterns, with each test method targeting a specific behavior or error condition.

```python

# From tests/test_run_evals.py — example test structure

import unittest
from scripts.run_evals import validate_cases, summarize_scores

class TestEvalHarness(unittest.TestCase):
    def test_validate_cases_rejects_duplicates(self):
        cases = [
            {"id": "case-1", "prompt": "test", "expected": "output"},
            {"id": "case-1", "prompt": "test2", "expected": "output2"},  # duplicate

        ]
        with self.assertRaises(ValueError) as ctx:
            validate_cases(cases)
        self.assertIn("duplicate", str(ctx.exception).lower())

```

Run the full suite from the repository root:

```bash
python -m unittest discover -s tests

```

## Temporary-File Isolation for Safe I/O Testing

File system operations are tested using `tempfile.TemporaryDirectory` to prevent side effects on the repository. This pattern appears throughout [`tests/test_run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/tests/test_run_evals.py) (lines 19–31) for simulating malformed inputs and runner configurations.

```python
import json
import tempfile
from pathlib import Path
import unittest
from scripts.run_evals import read_jsonl

class JsonlParsingTest(unittest.TestCase):
    def test_malformed_line_raises_clear_error(self):
        with tempfile.TemporaryDirectory() as td:
            path = Path(td) / "corrupt.jsonl"
            path.write_text(
                json.dumps({"id": "valid-case", "prompt": "test"}) + "\n"
                "this is not valid json\n"
            )
            
            with self.assertRaisesRegex(ValueError, "line 2"):
                read_jsonl(path)

```

This approach guarantees that:
- Tests remain hermetic and parallelizable
- No manual cleanup is required
- Error messages include precise line numbers for debugging

## Edge-Case and Error Condition Coverage

The test suite deliberately injects failure modes that production code must handle gracefully. Key scenarios verified include:

| Error Condition | Test Location | Validation Target |
|-----------------|-------------|-------------------|
| Duplicate case IDs | lines 65–90 | `validate_cases()` raises `ValueError` |
| Missing required fields | lines 101–119 | Schema validation catches omissions |
| Invalid JSON lines | lines 19–23, 24–31 | `read_jsonl()` provides line-specific errors |
| Unmetered runner rejection | lines 124–131 | Budget-aware runners only |
| Mismatched condition-row pairings | lines 148–164 | Baseline/candidate symmetry enforced |

Each assertion verifies both that an exception is raised and that the message contains actionable details for developers.

## CI Integration Testing with GitHub Actions

Beyond unit tests, the project validates end-to-end plugin functionality through two GitHub Actions workflows:

### Plugin Load Check ([`.github/workflows/plugin-load-check.yml`](https://github.com/ayghri/i-have-adhd/blob/main/.github/workflows/plugin-load-check.yml))

This job installs Claude Code in a fresh environment, registers the plugin, and confirms it reaches **"enabled"** status. It catches schema-validation errors, missing dependencies, or API incompatibilities that unit tests cannot detect.

```yaml

# Simplified structure of plugin-load-check.yml

jobs:
  verify-plugin-loads:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install Claude Code
        run: # ... installation steps

      - name: Register and verify plugin
        run: |
          claude plugin install .
          claude plugin status i-have-adhd | grep -q "enabled"

```

### Claude Trigger Workflow ([`.github/workflows/claude.yml`](https://github.com/ayghri/i-have-adhd/blob/main/.github/workflows/claude.yml))

Provides additional integration coverage by triggering Claude Code actions on issue comments, ensuring the plugin responds correctly to real interaction patterns.

Together, these workflows provide **continuous validation** that the plugin remains compatible with the evolving Claude runtime environment.

## Manual Evaluation Scaffolding

The repository includes [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) as a runnable evaluation harness. While not an automated test, it serves as a **developer sanity-check** for full pipeline execution:

```bash

# Manual evaluation run

python scripts/run_evals.py \
  --cases evals/cases.jsonl \
  --runner claude-3-opus \
  --output results/

```

CI jobs can invoke this script for smoke testing, and developers use it to debug evaluation logic before committing changes.

## Extending the Test Suite: Practical Patterns

### Adding a Custom Metric Test

When introducing new evaluation dimensions, extend both the `WEIGHTS` dictionary and the test assertions:

```python
import unittest
from scripts.run_evals import summarize_scores, WEIGHTS

class WeightedMetricTest(unittest.TestCase):
    def test_clarity_metric_integration(self):
        # Register new metric weight

        WEIGHTS["clarity"] = 0.05
        
        scores = [
            {
                "case_id": "test", "trial": 1, "condition": "baseline",
                "correctness": 4, "autonomy": 4, "actionability": 4,
                "safety": 4, "concision": 4, "clarity": 5,
                "blocker": False, "notes": "baseline run"
            },
            {
                "case_id": "test", "trial": 1, "condition": "candidate",
                "correctness": 5, "autonomy": 5, "actionability": 5,
                "safety": 5, "concision": 5, "clarity": 5,
                "blocker": False, "notes": "candidate run"
            },
        ]
        
        summary = summarize_scores(scores)
        self.assertIn("clarity", summary["conditions"]["baseline"])
        self.assertAlmostEqual(
            summary["conditions"]["baseline"]["clarity"], 
            5.0
        )
        
        # Cleanup to prevent cross-test pollution

        del WEIGHTS["clarity"]

```

Place this in `tests/` and run `python -m unittest discover -s tests` to verify integration.

## Key Testing Files Reference

| File | Purpose |
|------|---------|
| [`tests/test_run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/tests/test_run_evals.py) | Core unit-test suite with 15+ test methods covering validation, scoring, and error handling |
| [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) | Evaluation engine with functions `read_jsonl()`, `validate_cases()`, `summarize_scores()`, `run_evals()` |
| [`.github/workflows/plugin-load-check.yml`](https://github.com/ayghri/i-have-adhd/blob/main/.github/workflows/plugin-load-check.yml) | CI validation that plugin loads and enables in Claude Code |
| [`.github/workflows/claude.yml`](https://github.com/ayghri/i-have-adhd/blob/main/.github/workflows/claude.yml) | Integration workflow for comment-triggered Claude interactions |
| [`evals/README.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/README.md) | Documentation for evaluation data format, essential for writing valid test fixtures |

## Summary

- **Unit testing** uses Python's `unittest` framework in [`tests/test_run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/tests/test_run_evals.py) to verify individual functions and error pathways
- **Temporary-file isolation** ensures hermetic, parallel-safe I/O testing without repository pollution
- **Edge-case coverage** deliberately injects malformed inputs, duplicates, and schema violations to validate error handling
- **CI integration tests** via GitHub Actions confirm the plugin loads correctly in fresh Claude Code environments
- **Manual scaffolding** in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) supports exploratory testing and debugging

## Frequently Asked Questions

### What testing framework does i-have-adhd use?

The project uses Python's built-in **`unittest`** framework exclusively. No external test dependencies like `pytest` are required, keeping the repository lightweight and compatible with standard Python installations.

### How does the project test file I/O without side effects?

Tests use **`tempfile.TemporaryDirectory`** as a context manager to create isolated file system environments. This pattern appears in [`tests/test_run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/tests/test_run_evals.py) (lines 19–31) for simulating malformed JSON-L files and runner configurations, ensuring no test artifacts persist after execution.

### What integration testing does i-have-adhd perform?

Two GitHub Actions workflows provide integration coverage: **[`plugin-load-check.yml`](https://github.com/ayghri/i-have-adhd/blob/main/plugin-load-check.yml)** verifies the plugin installs and enables in a fresh Claude Code environment, while **[`claude.yml`](https://github.com/ayghri/i-have-adhd/blob/main/claude.yml)** tests comment-triggered interactions. These catch runtime incompatibilities that unit tests cannot detect.

### How can I add a new test for custom evaluation logic?

Create a new `unittest.TestCase` subclass in `tests/`, import functions from `scripts/run_evals`, and follow the existing patterns for temporary-file isolation and assertion styles. Run `python -m unittest discover -s tests` to validate your addition before submitting changes.