Understanding the Multi-Layered Verification Pipeline for I-Have-ADHD

The I-Have-ADHD project implements a rigorous seven-layer verification pipeline that chains automated checks for unit tests, schema validation, isolated execution, and release-gate scoring to ensure every pull request meets safety and correctness standards before merging.

The ayghri/i-have-adhd repository maintains high code quality through a comprehensive multi-layered verification pipeline. This system validates every change across multiple dimensions—ranging from Python unit tests to LLM evaluation scenarios—ensuring that modifications to skills, manifests, and evaluation cases are safe, reproducible, and performant.

The Seven Layers of Automated Verification

The pipeline consists of seven distinct validation layers, each targeting a specific aspect of code integrity. These layers are orchestrated through GitHub Actions and must all pass before a pull request becomes mergeable.

Layer 1: Unit-Test Validation

The foundation of the pipeline verifies core code behavior through Python’s unittest framework. This layer ensures that manifest files, skill metadata, and extension logic function as expected after changes.

The test suite resides in the tests/ directory and is executed via:

python3 -m unittest discover -s tests -v

Key test files like tests/test_omp_package.py validate package structures and metadata parsing, catching regressions in core utility functions before they reach the evaluation stage.

Layer 2: Evaluation-Case Schema Validation

Before any LLM evaluation occurs, the pipeline validates the structure of the evaluation catalog stored in evals/cases.jsonl. This schema check ensures all required fields—id, category, prompt, risk, and criteria—are present and that case IDs are unique.

The validation runs through:

python3 scripts/run_evals.py validate

Internally, this executes load_cases() to parse the JSONL file and validate_cases() (defined in scripts/run_evals.py lines 62–98) to enforce schema compliance, preventing malformed evaluation scenarios from entering the test matrix.

Layer 3: Paired-Run Planning

The third layer generates a deterministic execution matrix comparing baseline versus candidate conditions. This planning phase produces JSONL rows that dictate which cases to run, under which conditions, and for how many trials.

Execute the planning phase with:

python3 scripts/run_evals.py plan --trials 3 --include-comparator

The plan sub-command in scripts/run_evals.py (lines 90–100) emits structured output like {"case_id":"debugging-cause","trial":1,"condition":"baseline"}, ensuring reproducible test coverage across multiple experimental conditions.

Layer 4: Isolated Execution of Each Condition

Each evaluation runs in an isolated environment with strict budget enforcement. The run sub-command in scripts/run_evals.py establishes a neutral working directory via _neutral_cwd(), constructs prompts using _condition_prompt(), and executes the LLM runner through subprocess.run().

Run the candidate condition with:

python3 scripts/run_evals.py run \
    --runner my-runner \
    --condition candidate \
    --condition-skill skills/i-have-adhd/SKILL.md \
    --budget-usd 5 \
    --output evals/results/candidate.jsonl

This layer tracks raw responses, token usage, and costs, writing one JSON line per trial to the output file while respecting the --budget-usd constraint to prevent runaway spending during testing.

Layer 5: Scoring and Release-Gate Logic

After execution, the pipeline aggregates human-judged scores against the weighted rubric defined in evals/rubric.md. The summarize_scores() function in scripts/run_evals.py (lines 148–186) calculates weighted metrics and determines whether the candidate passes the release gate.

Score results using:

python3 scripts/run_evals.py score evals/results/candidate.jsonl

The release gate fails if any blocking safety or correctness findings exist, if regression exceeds 0.1 points, or if the weighted candidate score does not exceed the baseline. This ensures only improvements that meet safety thresholds are promoted.

Layer 6: Hook and Manifest Verification

Platform-specific integrations are validated through dedicated CI workflows that load manifests without errors. The workflows .github/workflows/plugin-load-check.yml and .github/workflows/pi-load-check.yml execute npm run or bun run commands against plugin.json, *.extension.json, and hooks/hooks.json files.

These checks confirm that always-on hooks (implemented in hooks/always-on.* for shell, PowerShell, and Node.js) load correctly across supported platforms, preventing runtime failures in production environments.

Layer 7: CI Orchestration

The final layer coordinates all previous checks through .github/workflows/ci.yml. This workflow runs the unit tests, evaluation validation, and hook verification in sequence, aborting the pipeline immediately upon any failure.

The CI configuration ensures that verification is automatic and mandatory, blocking merges until the candidate passes the unit-test suite, maintains a valid evaluation catalog, executes correctly under budget constraints, scores above baseline, and loads all platform manifests.

How the Pipeline Works in Practice

Contributors interact with the pipeline through a structured command sequence that mirrors the CI execution:

1. Validate local changes against the test suite and schema requirements
2. Plan the evaluation matrix to determine required test runs
3. Execute conditions in isolated environments with cost controls
4. Score results against the weighted rubric to verify release readiness

Each command maps to specific functions in scripts/run_evals.py, providing transparent validation feedback before submitting pull requests.

Summary

The I-Have-ADHD verification pipeline ensures code quality through:

• Unit-test coverage via tests/ and Python’s unittest framework
• Schema enforcement for evals/cases.jsonl through validate_cases()
• Deterministic planning that generates baseline/candidate run matrices
• Isolated execution with budget enforcement via _condition_prompt() and _neutral_cwd()
• Weighted scoring against evals/rubric.md with automated release-gate decisions
• Platform manifest validation through GitHub Actions workflows
• Mandatory CI orchestration that blocks merges on any failure

Frequently Asked Questions

What triggers the multi-layered verification pipeline?

Any pull request targeting the main branch automatically triggers the full pipeline through .github/workflows/ci.yml. The workflow runs all seven layers sequentially, from unit tests in tests/ through hook verification in .github/workflows/plugin-load-check.yml, ensuring comprehensive validation before merge.

How does the pipeline prevent expensive LLM evaluation errors?

The run_evaluations() function in scripts/run_evals.py enforces cost controls through the --budget-usd parameter, which caps spending per evaluation run. Additionally, the planning layer validates the case catalog schema beforehand, preventing malformed inputs from triggering unnecessary LLM calls.

Where is the release-gate logic defined?

The release-gate criteria reside in evals/rubric.md and are implemented in the summarize_scores() function within scripts/run_evals.py (lines 148–186). The logic blocks releases upon detecting blocking safety findings, correctness regressions exceeding 0.1 points, or candidate scores failing to exceed baseline performance.

Can contributors run the verification pipeline locally?

Yes. Contributors can execute the complete validation sequence locally using the commands listed in CONTRIBUTING.md and AGENTS.md. The scripts/run_evals.py script provides sub-commands for validate, plan, run, and score, enabling full pipeline simulation before submitting pull requests.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}