# Understanding the Multi-Layered Verification Pipeline for I-Have-ADHD

> Explore the seven-layer verification pipeline for I-Have-ADHD. Discover how automated checks for unit tests, schema validation, and more ensure pull request safety and correctness.

- Repository: [Ayoub Ghriss/i-have-adhd](https://github.com/ayghri/i-have-adhd)
- Tags: deep-dive
- Published: 2026-09-09

---

**The I-Have-ADHD project implements a rigorous seven-layer verification pipeline that chains automated checks for unit tests, schema validation, isolated execution, and release-gate scoring to ensure every pull request meets safety and correctness standards before merging.**

The **ayghri/i-have-adhd** repository maintains high code quality through a comprehensive multi-layered verification pipeline. This system validates every change across multiple dimensions—ranging from Python unit tests to LLM evaluation scenarios—ensuring that modifications to skills, manifests, and evaluation cases are safe, reproducible, and performant.

## The Seven Layers of Automated Verification

The pipeline consists of seven distinct validation layers, each targeting a specific aspect of code integrity. These layers are orchestrated through GitHub Actions and must all pass before a pull request becomes mergeable.

### Layer 1: Unit-Test Validation

The foundation of the pipeline verifies core code behavior through Python’s unittest framework. This layer ensures that manifest files, skill metadata, and extension logic function as expected after changes.

The test suite resides in the `tests/` directory and is executed via:

```bash
python3 -m unittest discover -s tests -v

```

Key test files like [`tests/test_omp_package.py`](https://github.com/ayghri/i-have-adhd/blob/main/tests/test_omp_package.py) validate package structures and metadata parsing, catching regressions in core utility functions before they reach the evaluation stage.

### Layer 2: Evaluation-Case Schema Validation

Before any LLM evaluation occurs, the pipeline validates the structure of the evaluation catalog stored in `evals/cases.jsonl`. This schema check ensures all required fields—`id`, `category`, `prompt`, `risk`, and `criteria`—are present and that case IDs are unique.

The validation runs through:

```bash
python3 scripts/run_evals.py validate

```

Internally, this executes `load_cases()` to parse the JSONL file and `validate_cases()` (defined in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) lines 62–98) to enforce schema compliance, preventing malformed evaluation scenarios from entering the test matrix.

### Layer 3: Paired-Run Planning

The third layer generates a deterministic execution matrix comparing **baseline** versus **candidate** conditions. This planning phase produces JSONL rows that dictate which cases to run, under which conditions, and for how many trials.

Execute the planning phase with:

```bash
python3 scripts/run_evals.py plan --trials 3 --include-comparator

```

The `plan` sub-command in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) (lines 90–100) emits structured output like `{"case_id":"debugging-cause","trial":1,"condition":"baseline"}`, ensuring reproducible test coverage across multiple experimental conditions.

### Layer 4: Isolated Execution of Each Condition

Each evaluation runs in an isolated environment with strict budget enforcement. The `run` sub-command in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) establishes a neutral working directory via `_neutral_cwd()`, constructs prompts using `_condition_prompt()`, and executes the LLM runner through `subprocess.run()`.

Run the candidate condition with:

```bash
python3 scripts/run_evals.py run \
    --runner my-runner \
    --condition candidate \
    --condition-skill skills/i-have-adhd/SKILL.md \
    --budget-usd 5 \
    --output evals/results/candidate.jsonl

```

This layer tracks raw responses, token usage, and costs, writing one JSON line per trial to the output file while respecting the `--budget-usd` constraint to prevent runaway spending during testing.

### Layer 5: Scoring and Release-Gate Logic

After execution, the pipeline aggregates human-judged scores against the weighted rubric defined in [`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md). The `summarize_scores()` function in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) (lines 148–186) calculates weighted metrics and determines whether the candidate passes the release gate.

Score results using:

```bash
python3 scripts/run_evals.py score evals/results/candidate.jsonl

```

The release gate fails if any blocking safety or correctness findings exist, if regression exceeds 0.1 points, or if the weighted candidate score does not exceed the baseline. This ensures only improvements that meet safety thresholds are promoted.

### Layer 6: Hook and Manifest Verification

Platform-specific integrations are validated through dedicated CI workflows that load manifests without errors. The workflows [`.github/workflows/plugin-load-check.yml`](https://github.com/ayghri/i-have-adhd/blob/main/.github/workflows/plugin-load-check.yml) and [`.github/workflows/pi-load-check.yml`](https://github.com/ayghri/i-have-adhd/blob/main/.github/workflows/pi-load-check.yml) execute `npm run` or `bun run` commands against [`plugin.json`](https://github.com/ayghri/i-have-adhd/blob/main/plugin.json), `*.extension.json`, and [`hooks/hooks.json`](https://github.com/ayghri/i-have-adhd/blob/main/hooks/hooks.json) files.

These checks confirm that always-on hooks (implemented in `hooks/always-on.*` for shell, PowerShell, and Node.js) load correctly across supported platforms, preventing runtime failures in production environments.

### Layer 7: CI Orchestration

The final layer coordinates all previous checks through [`.github/workflows/ci.yml`](https://github.com/ayghri/i-have-adhd/blob/main/.github/workflows/ci.yml). This workflow runs the unit tests, evaluation validation, and hook verification in sequence, aborting the pipeline immediately upon any failure.

The CI configuration ensures that verification is automatic and mandatory, blocking merges until the candidate passes the unit-test suite, maintains a valid evaluation catalog, executes correctly under budget constraints, scores above baseline, and loads all platform manifests.

## How the Pipeline Works in Practice

Contributors interact with the pipeline through a structured command sequence that mirrors the CI execution:

1. **Validate local changes** against the test suite and schema requirements
2. **Plan the evaluation matrix** to determine required test runs
3. **Execute conditions** in isolated environments with cost controls
4. **Score results** against the weighted rubric to verify release readiness

Each command maps to specific functions in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py), providing transparent validation feedback before submitting pull requests.

## Summary

The I-Have-ADHD verification pipeline ensures code quality through:

- **Unit-test coverage** via `tests/` and Python’s unittest framework
- **Schema enforcement** for `evals/cases.jsonl` through `validate_cases()`
- **Deterministic planning** that generates baseline/candidate run matrices
- **Isolated execution** with budget enforcement via `_condition_prompt()` and `_neutral_cwd()`
- **Weighted scoring** against [`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md) with automated release-gate decisions
- **Platform manifest validation** through GitHub Actions workflows
- **Mandatory CI orchestration** that blocks merges on any failure

## Frequently Asked Questions

### What triggers the multi-layered verification pipeline?

Any pull request targeting the main branch automatically triggers the full pipeline through [`.github/workflows/ci.yml`](https://github.com/ayghri/i-have-adhd/blob/main/.github/workflows/ci.yml). The workflow runs all seven layers sequentially, from unit tests in `tests/` through hook verification in [`.github/workflows/plugin-load-check.yml`](https://github.com/ayghri/i-have-adhd/blob/main/.github/workflows/plugin-load-check.yml), ensuring comprehensive validation before merge.

### How does the pipeline prevent expensive LLM evaluation errors?

The `run_evaluations()` function in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) enforces cost controls through the `--budget-usd` parameter, which caps spending per evaluation run. Additionally, the planning layer validates the case catalog schema beforehand, preventing malformed inputs from triggering unnecessary LLM calls.

### Where is the release-gate logic defined?

The release-gate criteria reside in [`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md) and are implemented in the `summarize_scores()` function within [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) (lines 148–186). The logic blocks releases upon detecting blocking safety findings, correctness regressions exceeding 0.1 points, or candidate scores failing to exceed baseline performance.

### Can contributors run the verification pipeline locally?

Yes. Contributors can execute the complete validation sequence locally using the commands listed in [`CONTRIBUTING.md`](https://github.com/ayghri/i-have-adhd/blob/main/CONTRIBUTING.md) and [`AGENTS.md`](https://github.com/ayghri/i-have-adhd/blob/main/AGENTS.md). The [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) script provides sub-commands for `validate`, `plan`, `run`, and `score`, enabling full pipeline simulation before submitting pull requests.