# How the i-have-adhd Skill Ensures Accuracy Through Automated Testing

> Discover how the i-have-adhd skill ensures accuracy with automated testing. This article details the unit-test suite that validates case catalogs, scoring logic, blocker detection, and budget safeguards for reliable performance.

- Repository: [Ayoub Ghriss/i-have-adhd](https://github.com/ayghri/i-have-adhd)
- Tags: testing
- Published: 2026-08-03

---

**The ayghri/i-have-adhd repository guarantees i-have-adhd skill accuracy by running a comprehensive unit-test suite in [`tests/test_run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/tests/test_run_evals.py) that validates case catalogs, scoring logic, blocker detection, and budget safeguards.**

The evaluation pipeline for the i-have-adhd skill relies on rigorous automated testing to prevent regressions and ensure consistent behavior. Implemented in the `ayghri/i-have-adhd` repository, the testing framework validates every component of the evaluation workflow—from JSONL parsing to release gate decisions—ensuring that only high-quality, safe model outputs pass through the pipeline.

## Validating Evaluation Case Integrity

The foundation of accurate skill assessment begins with the case catalog stored in `evals/cases.jsonl`. The test suite ensures these evaluation cases meet strict structural requirements before any scoring occurs.

### Case Catalog Validation

The `test_case_catalog_is_valid_and_balanced` function in [`tests/test_run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/tests/test_run_evals.py) (lines 15-23) loads cases using `run_evals.validate_cases` to verify required fields, unique IDs, proper risk levels, and non-empty criteria. This validation confirms that the evaluation dataset contains a minimum number of cases across diverse categories, preventing skewed results from an incomplete test suite.

### Duplicate Detection

To maintain data integrity, the framework rejects duplicate entries aggressively. The tests `test_duplicate_score_rows_are_rejected` (lines 81-88) and `test_duplicate_case_ids_are_rejected` (lines 106-115) verify that the validation logic raises appropriate errors when duplicate case IDs or score rows appear in the dataset, ensuring each evaluation trial represents a unique scenario.

## Scoring Logic and Release Gates

Beyond data validation, the testing framework verifies that score aggregation and release decisions function correctly under various conditions.

### Weighted Score Calculation

The `test_score_summary_applies_weights_and_release_gates` test (lines 24-47) validates the `run_evals.summarize_scores` function. By creating synthetic score rows for baseline and candidate conditions, the test confirms that weighted scores calculate correctly and that release gates pass only when quality thresholds are met.

### Blocker Detection and Safety

Critical safety regressions are caught through `test_candidate_blocker_fails_release_gate` (lines 48-70). This test supplies a blocking finding for the candidate condition and asserts that the release gate fails when the `blocker` flag is present, ensuring that safety-critical issues prevent automatic deployment regardless of other metrics.

## Data Consistency and Parsing Robustness

The testing framework enforces strict consistency between evaluation conditions and robust handling of input files.

### Pairing Consistency

Evaluation fairness requires that baseline and candidate conditions are judged on identical case-trial pairs. The `test_conditions_judged_on_different_cases_are_rejected` test (lines 71-79) feeds mismatched rows into the scoring function and expects a `ValueError`, guaranteeing that any deviation in evaluation pairs triggers an immediate error.

### JSONL Parsing Validation

The `run_evals.read_jsonl` function undergoes validation through `test_jsonl_loader_reports_invalid_rows` (lines 17-23). By writing malformed JSONL files and confirming that `ValueError` is raised with correct line numbers, the test ensures that parsing errors are caught early with precise diagnostic information.

## Budgeting and Cost Controls

The evaluation pipeline includes safeguards against unintended API costs.

### Metering Safeguards

The `test_unmetered_runner_is_rejected_before_any_call` test (lines 24-62) verifies that the runner refuses to execute unmetered requests unless the `--allow-unmetered` flag is explicitly set. This prevents hidden cost regressions by validating budget constraints before any external API calls occur.

## Continuous Integration

The repository automates regression detection through GitHub Actions. The workflow defined in [`.github/workflows/plugin-load-check.yml`](https://github.com/ayghri/i-have-adhd/blob/main/.github/workflows/plugin-load-check.yml) executes the full test suite on every push, providing immediate feedback on any changes that might compromise the i-have-adhd skill's evaluation accuracy.

## Running the Tests Locally

Developers can execute the validation suite using standard Python unittest commands:

```bash
python -m unittest discover -s tests

```

To programmatically validate a case catalog:

```python
from scripts.run_evals import load_cases, validate_cases
from pathlib import Path

cases = load_cases(Path("evals/cases.jsonl"))
errors = validate_cases(cases)
assert not errors, f"Catalog errors: {errors}"

```

For manual score summarization:

```python
from scripts.run_evals import read_jsonl, summarize_scores

scores = read_jsonl(Path("my_scores.jsonl"))
summary = summarize_scores(scores)
print(summary["release_gate"]["passed"])   # True ⇔ candidate passes baseline

```

## Summary

- **Case catalog validation** ensures every evaluation case has required fields, unique IDs, and proper risk levels through `validate_cases` in [`tests/test_run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/tests/test_run_evals.py) (lines 15-23).
- **Scoring logic verification** confirms weighted calculations and release gate behavior via `test_score_summary_applies_weights_and_release_gates` (lines 24-47).
- **Blocker detection** prevents unsafe releases through `test_candidate_blocker_fails_release_gate` (lines 48-70).
- **Data consistency checks** enforce identical case-trial pairs across conditions and reject duplicates (lines 71-79, 81-88, and 106-115).
- **Robust parsing** guarantees malformed JSONL files are caught with precise line numbers (lines 17-23).
- **Budget safeguards** protect against unmetered API calls unless explicitly authorized (lines 24-62).
- **Continuous integration** runs the full suite automatically via [`.github/workflows/plugin-load-check.yml`](https://github.com/ayghri/i-have-adhd/blob/main/.github/workflows/plugin-load-check.yml).

## Frequently Asked Questions

### How does the i-have-adhd skill prevent duplicate evaluation cases?

The testing framework explicitly rejects duplicates through `test_duplicate_case_ids_are_rejected` and `test_duplicate_score_rows_are_rejected` in [`tests/test_run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/tests/test_run_evals.py) (lines 81-88 and 106-115). These tests validate that the `validate_cases` function raises appropriate errors when duplicate IDs or score rows appear, ensuring each evaluation represents a unique scenario.

### What happens if the baseline and candidate conditions use different case sets?

The `test_conditions_judged_on_different_cases_are_rejected` test (lines 71-79) verifies that the `summarize_scores` function raises a `ValueError` when case-trial pairs mismatch between conditions. This ensures evaluation fairness by requiring identical test sets for both baseline and candidate comparisons.

### How does the repository prevent accidental API costs during evaluation?

The `test_unmetered_runner_is_rejected_before_any_call` test (lines 24-62) enforces budget controls by verifying that the evaluation runner refuses to execute unless the `--allow-unmetered` flag is present. This metering safeguard prevents hidden cost regressions by blocking unmetered requests before any external API calls occur.

### Where is the evaluation logic implemented in the i-have-adhd repository?

The core evaluation engine resides in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py), which handles case loading, validation, runner execution, response parsing, and score aggregation. The corresponding unit tests in [`tests/test_run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/tests/test_run_evals.py) verify each component, while `evals/cases.jsonl` stores the evaluation catalog and [`.github/workflows/plugin-load-check.yml`](https://github.com/ayghri/i-have-adhd/blob/main/.github/workflows/plugin-load-check.yml) automates continuous validation.