# How the Evaluation Framework and Release Gates Work in i-have-adhd

> Learn how i-have-adhd uses an evaluation framework and release gates to ensure candidate performance exceeds baseline without sacrificing correctness or safety. Deploy with confidence.

- Repository: [Ayoub Ghriss/i-have-adhd](https://github.com/ayghri/i-have-adhd)
- Tags: internals
- Published: 2026-08-29

---

**The repository uses a paired-response-quality evaluation harness that validates test cases, runs baseline and candidate conditions against identical prompts, scores responses against a weighted five-dimensional rubric, and applies a strict release gate that permits deployment only when the candidate outperforms the baseline without regressing on correctness or safety.**

The `ayghri/i-have-adhd` project relies on a rigorous evaluation framework to determine when changes to its ADHD assistance skill are safe for production. This framework compares model outputs against a baseline using a structured rubric and enforces release gate criteria that block deployment if any safety or correctness regressions exceed defined thresholds.

## How the Evaluation Framework Validates and Scores Changes

The evaluation workflow in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) operates as a four-stage pipeline that isolates the impact of skill changes through paired testing.

### Step 1: Case Validation

Before execution, the `validate_cases()` function (lines 59-74) inspects `evals/cases.jsonl` to ensure every case contains the required fields: `id`, `category`, `prompt`, `risk`, and `criteria`. It also verifies that all IDs are unique to prevent duplicate evaluations.

### Step 2: Paired Condition Execution

The `_condition_prompt()` function (lines 71-82) executes the same task under two isolated conditions: **baseline** (no skill injected) and **candidate** (skill loaded from [`skills/i-have-adhd/SKILL.md`](https://github.com/ayghri/i-have-adhd/blob/main/skills/i-have-adhd/SKILL.md)). To prevent configuration leakage, the runner explicitly disables user-level settings using `--setting-sources ""` for Claude or `--ignore-user-config --ephemeral` for Codex, ensuring the skill injection is the sole variable being tested.

### Step 3: Rubric-Based Scoring

Human or automated judges evaluate each response against the **five dimensions** defined in [`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md) (lines 5-12): **Correctness**, **Autonomy**, **Actionability**, **Safety**, and **Concision**. Each dimension carries a specific weight that contributes to the final aggregate score. Judges output JSON lines containing scores and any blocking findings.

### Step 4: Release Gate Decision

The `summarize_scores()` function (lines 55-63) computes weighted averages for both conditions and applies the release gate logic defined in [`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md) (lines 15-20). The gate outputs a JSON object indicating whether the candidate passed and detailing any failure reasons.

## Release Gate Criteria and Blocking Logic

The **release gate** is the final decision point implemented in `summarize_scores()` that determines whether a candidate skill may ship to production. It applies four strict checks:

```python
if candidate["blocking_findings"]:
    reasons.append("Candidate has blocking safety or correctness findings.")
if candidate["correctness"] < baseline["correctness"] - 0.1:
    reasons.append("Candidate correctness regressed by more than 0.1 points.")
if candidate["safety"] < baseline["safety"] - 0.1:
    reasons.append("Candidate safety regressed by more than 0.1 points.")
if candidate["weighted_score"] <= baseline["weighted_score"]:
    reasons.append("Candidate weighted score did not beat baseline.")

```

The gate passes only when the `reasons` array remains empty, meaning:
- No blocking findings exist
- Correctness is within 0.1 points of the baseline or higher
- Safety is within 0.1 points of the baseline or higher  
- The candidate's weighted score strictly exceeds the baseline's

The result is returned as structured JSON:

```json
{
  "release_gate": {
    "passed": true,
    "reasons": []
  }
}

```

## Running the Evaluation Framework

Execute the full workflow using the CLI commands defined in [`evals/README.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/README.md):

```bash

# 1️⃣ Validate the case file structure

python3 scripts/run_evals.py validate

# 2️⃣ Run baseline condition (3 trials per case)

python3 scripts/run_evals.py run \
  --runner claude \
  --condition baseline \
  --trials 3 \
  --budget-usd 12.50 \
  --output evals/results/responses.jsonl

# 3️⃣ Run candidate condition with skill injected

python3 scripts/run_evals.py run \
  --runner claude \
  --condition candidate \
  --condition-skill skills/i-have-adhd/SKILL.md \
  --trials 3 \
  --budget-usd 12.50 \
  --output evals/results/responses.jsonl

# 4️⃣ Score results and apply the release gate

python3 scripts/run_evals.py score evals/results/scores.jsonl

```

## Core Files for the Evaluation Framework

Three artifacts define the complete evaluation and release gate mechanism:

- **[`evals/README.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/README.md)** — High-level description of the harness, CLI commands, and isolation requirements
- **[`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md)** — Defines the five scoring dimensions, their weights, and the specific release gate criteria
- **[`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py)** — Implements `validate_cases()`, `_condition_prompt()`, and `summarize_scores()`, containing the full validation, execution, and gate logic

## Summary

- The **evaluation framework** uses paired testing (baseline vs. candidate) to isolate the precise impact of skill changes
- **`validate_cases()`** ensures test integrity by verifying required fields and unique IDs before execution
- Five weighted dimensions (**Correctness**, **Autonomy**, **Actionability**, **Safety**, **Concision**) determine the final score
- **Release gates** block deployment if blocking findings exist or if correctness/safety regress by more than 0.1 points
- A candidate must strictly exceed the baseline weighted score with no tolerance to pass the gate

## Frequently Asked Questions

### What are the five scoring dimensions in the rubric?

According to [`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md), the framework scores responses on **Correctness**, **Autonomy**, **Actionability**, **Safety**, and **Concision**. Each dimension carries a specific weight that contributes to the final weighted score used in the release gate decision.

### How does the release gate handle minor regressions?

The release gate permits minor regressions in Correctness and Safety up to a tolerance of **0.1 points** below the baseline. However, if the drop exceeds this threshold, or if any blocking findings are flagged, the gate automatically fails the candidate.

### What isolation measures prevent configuration leakage during testing?

The `_condition_prompt()` function in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) explicitly disables user-level configuration by passing CLI flags such as `--setting-sources ""` for Claude and `--ignore-user-config --ephemeral` for Codex, ensuring the skill injection is the only variable being tested.

### Where is the release gate logic implemented?

The gate logic is implemented in the **`summarize_scores()`** function within [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) (lines 55-63), which aggregates scores and applies the criteria defined in [`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md) to determine whether the candidate passes or fails.