How to Run Tests for the i-have-adhd Project: A Complete Guide

Run python -m unittest discover -s tests from the repository root to execute the 9 test cases in tests/test_run_evals.py that validate the evaluation harness.

The ayghri/i-have-adhd repository ships with a built-in test suite that validates the evaluation harness implemented in scripts/run_evals.py. These tests use Python’s standard unittest framework, requiring no third-party dependencies or complex configuration. Running the test suite ensures that case loading, score aggregation, and release gate logic function correctly before deploying the evaluation pipeline.

Prerequisites

Before executing tests, ensure you have Python 3.9 or newer installed, as the codebase utilizes modern type hints incompatible with earlier versions. Clone the repository and navigate to the root directory containing the scripts/ and tests/ subdirectories.

Running the Full Test Suite

To execute all tests at once, use Python’s unittest discovery mechanism from the repository root:

python -m unittest discover -s tests

This command automatically locates and runs all test_*.py files in the tests/ directory. Successful execution displays confirmation that 9 tests passed:

.........
----------------------------------------------------------------------
Ran 9 tests in 0.12s

OK

For verbose output showing individual test names and docstrings, append the -v flag:

python -m unittest -v discover -s tests

Running Individual Test Files

To focus on specific functionality without running the entire suite, target a single test file directly:

python -m unittest tests/test_run_evals.py

This executes only the evaluation harness tests, providing faster feedback when debugging specific validation logic.

What the Tests Validate

The test suite in tests/test_run_evals.py exercises nine critical validation scenarios covering the core utilities in scripts/run_evals.py:

• test_case_catalog_is_valid_and_balanced – Loads evals/cases.jsonl and verifies the file is well-formed JSONL containing a diverse set of evaluation cases.

• test_score_summary_applies_weights_and_release_gates – Calls run_evals.summarize_scores with minimal score rows to validate weighted score calculations and release-gate logic.

• test_candidate_blocker_fails_release_gate – Ensures that blocker flags in candidate rows correctly force the release gate to fail.

• test_conditions_judged_on_different_cases_are_rejected – Verifies that the summarizer raises an error when baseline and candidate rows reference different evaluation cases.

• test_duplicate_score_rows_are_rejected – Confirms that duplicate rows in score data raise a ValueError.

• test_duplicate_case_ids_are_rejected – Validates that the case validator catches duplicate id fields within the case catalog.

• test_jsonl_loader_reports_invalid_rows – Confirms that run_evals.read_jsonl raises clear errors when encountering malformed JSON lines.

• test_unmetered_runner_is_rejected_before_any_call – Ensures the runner aborts early when unmetered calls are not allowed, preventing unnecessary API invocations.

• test_completed_keys_support_resuming_partial_runs – Tests the helper function that tracks which evaluation rows have already been written, enabling resumable evaluation runs.

Key Files and Functions Tested

The tests validate the evaluation pipeline in scripts/run_evals.py, which implements:

• Case management – Loading and validating entries from evals/cases.jsonl
• Prompt building – Constructing evaluation prompts for LLM runners
• Score aggregation – Weighted scoring calculations via summarize_scores
• Data parsing – Robust JSONL reading through read_jsonl
• CLI commands – Indirect testing of the validate, plan, score, and run subcommands

Summary

• The i-have-adhd project uses Python’s built-in unittest framework with no external test runners required.
• Run python -m unittest discover -s tests from the repository root to execute all 9 tests.
• Individual test files can be targeted with python -m unittest tests/test_run_evals.py.
• Tests validate scripts/run_evals.py functionality including case loading, scoring logic, and error handling.
• All tests complete in approximately 0.12 seconds, providing rapid feedback during development.

Frequently Asked Questions

Do I need to install pytest or other testing frameworks to run these tests?

No. The repository uses Python’s standard library unittest module exclusively. You can execute all tests using the built-in test runner without installing third-party packages like pytest, nose, or unittest2.

What Python version is required to run the test suite?

Python 3.9 or newer is required. The codebase utilizes type hints and syntax features introduced in recent Python versions, and running the tests on older versions will likely result in syntax errors or import failures when loading scripts/run_evals.py.

Why do I need to run the tests from the repository root?

The tests reference files using relative paths (such as evals/cases.jsonl and scripts/run_evals.py). Running from the root directory ensures these path references resolve correctly when the test loader imports modules and the test cases access data files on disk.

Can I test the CLI commands directly instead of using unittest?

While the repository provides a run_evals CLI with validate, plan, score, and run subcommands, the recommended approach is running the unittest suite. The tests indirectly exercise these CLI functions through their exposed Python APIs, providing faster and more isolated feedback than full CLI integration testing.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}