How the i-have-adhd Skill Ensures Accuracy Through Automated Testing

The ayghri/i-have-adhd repository guarantees i-have-adhd skill accuracy by running a comprehensive unit-test suite in tests/test_run_evals.py that validates case catalogs, scoring logic, blocker detection, and budget safeguards.

The evaluation pipeline for the i-have-adhd skill relies on rigorous automated testing to prevent regressions and ensure consistent behavior. Implemented in the ayghri/i-have-adhd repository, the testing framework validates every component of the evaluation workflow—from JSONL parsing to release gate decisions—ensuring that only high-quality, safe model outputs pass through the pipeline.

Validating Evaluation Case Integrity

The foundation of accurate skill assessment begins with the case catalog stored in evals/cases.jsonl. The test suite ensures these evaluation cases meet strict structural requirements before any scoring occurs.

Case Catalog Validation

The test_case_catalog_is_valid_and_balanced function in tests/test_run_evals.py (lines 15-23) loads cases using run_evals.validate_cases to verify required fields, unique IDs, proper risk levels, and non-empty criteria. This validation confirms that the evaluation dataset contains a minimum number of cases across diverse categories, preventing skewed results from an incomplete test suite.

Duplicate Detection

To maintain data integrity, the framework rejects duplicate entries aggressively. The tests test_duplicate_score_rows_are_rejected (lines 81-88) and test_duplicate_case_ids_are_rejected (lines 106-115) verify that the validation logic raises appropriate errors when duplicate case IDs or score rows appear in the dataset, ensuring each evaluation trial represents a unique scenario.

Scoring Logic and Release Gates

Beyond data validation, the testing framework verifies that score aggregation and release decisions function correctly under various conditions.

Weighted Score Calculation

The test_score_summary_applies_weights_and_release_gates test (lines 24-47) validates the run_evals.summarize_scores function. By creating synthetic score rows for baseline and candidate conditions, the test confirms that weighted scores calculate correctly and that release gates pass only when quality thresholds are met.

Blocker Detection and Safety

Critical safety regressions are caught through test_candidate_blocker_fails_release_gate (lines 48-70). This test supplies a blocking finding for the candidate condition and asserts that the release gate fails when the blocker flag is present, ensuring that safety-critical issues prevent automatic deployment regardless of other metrics.

Data Consistency and Parsing Robustness

The testing framework enforces strict consistency between evaluation conditions and robust handling of input files.

Pairing Consistency

Evaluation fairness requires that baseline and candidate conditions are judged on identical case-trial pairs. The test_conditions_judged_on_different_cases_are_rejected test (lines 71-79) feeds mismatched rows into the scoring function and expects a ValueError, guaranteeing that any deviation in evaluation pairs triggers an immediate error.

JSONL Parsing Validation

The run_evals.read_jsonl function undergoes validation through test_jsonl_loader_reports_invalid_rows (lines 17-23). By writing malformed JSONL files and confirming that ValueError is raised with correct line numbers, the test ensures that parsing errors are caught early with precise diagnostic information.

Budgeting and Cost Controls

The evaluation pipeline includes safeguards against unintended API costs.

Metering Safeguards

The test_unmetered_runner_is_rejected_before_any_call test (lines 24-62) verifies that the runner refuses to execute unmetered requests unless the --allow-unmetered flag is explicitly set. This prevents hidden cost regressions by validating budget constraints before any external API calls occur.

Continuous Integration

The repository automates regression detection through GitHub Actions. The workflow defined in .github/workflows/plugin-load-check.yml executes the full test suite on every push, providing immediate feedback on any changes that might compromise the i-have-adhd skill's evaluation accuracy.

Running the Tests Locally

Developers can execute the validation suite using standard Python unittest commands:

python -m unittest discover -s tests

To programmatically validate a case catalog:

from scripts.run_evals import load_cases, validate_cases
from pathlib import Path

cases = load_cases(Path("evals/cases.jsonl"))
errors = validate_cases(cases)
assert not errors, f"Catalog errors: {errors}"

For manual score summarization:

from scripts.run_evals import read_jsonl, summarize_scores

scores = read_jsonl(Path("my_scores.jsonl"))
summary = summarize_scores(scores)
print(summary["release_gate"]["passed"])   # True ⇔ candidate passes baseline

Summary

• Case catalog validation ensures every evaluation case has required fields, unique IDs, and proper risk levels through validate_cases in tests/test_run_evals.py (lines 15-23).
• Scoring logic verification confirms weighted calculations and release gate behavior via test_score_summary_applies_weights_and_release_gates (lines 24-47).
• Blocker detection prevents unsafe releases through test_candidate_blocker_fails_release_gate (lines 48-70).
• Data consistency checks enforce identical case-trial pairs across conditions and reject duplicates (lines 71-79, 81-88, and 106-115).
• Robust parsing guarantees malformed JSONL files are caught with precise line numbers (lines 17-23).
• Budget safeguards protect against unmetered API calls unless explicitly authorized (lines 24-62).
• Continuous integration runs the full suite automatically via .github/workflows/plugin-load-check.yml.

Frequently Asked Questions

How does the i-have-adhd skill prevent duplicate evaluation cases?

The testing framework explicitly rejects duplicates through test_duplicate_case_ids_are_rejected and test_duplicate_score_rows_are_rejected in tests/test_run_evals.py (lines 81-88 and 106-115). These tests validate that the validate_cases function raises appropriate errors when duplicate IDs or score rows appear, ensuring each evaluation represents a unique scenario.

What happens if the baseline and candidate conditions use different case sets?

The test_conditions_judged_on_different_cases_are_rejected test (lines 71-79) verifies that the summarize_scores function raises a ValueError when case-trial pairs mismatch between conditions. This ensures evaluation fairness by requiring identical test sets for both baseline and candidate comparisons.

How does the repository prevent accidental API costs during evaluation?

The test_unmetered_runner_is_rejected_before_any_call test (lines 24-62) enforces budget controls by verifying that the evaluation runner refuses to execute unless the --allow-unmetered flag is present. This metering safeguard prevents hidden cost regressions by blocking unmetered requests before any external API calls occur.

Where is the evaluation logic implemented in the i-have-adhd repository?

The core evaluation engine resides in scripts/run_evals.py, which handles case loading, validation, runner execution, response parsing, and score aggregation. The corresponding unit tests in tests/test_run_evals.py verify each component, while evals/cases.jsonl stores the evaluation catalog and .github/workflows/plugin-load-check.yml automates continuous validation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}