How the Evaluation Framework and Release Gates Work in i-have-adhd

The repository uses a paired-response-quality evaluation harness that validates test cases, runs baseline and candidate conditions against identical prompts, scores responses against a weighted five-dimensional rubric, and applies a strict release gate that permits deployment only when the candidate outperforms the baseline without regressing on correctness or safety.

The ayghri/i-have-adhd project relies on a rigorous evaluation framework to determine when changes to its ADHD assistance skill are safe for production. This framework compares model outputs against a baseline using a structured rubric and enforces release gate criteria that block deployment if any safety or correctness regressions exceed defined thresholds.

How the Evaluation Framework Validates and Scores Changes

The evaluation workflow in scripts/run_evals.py operates as a four-stage pipeline that isolates the impact of skill changes through paired testing.

Step 1: Case Validation

Before execution, the validate_cases() function (lines 59-74) inspects evals/cases.jsonl to ensure every case contains the required fields: id, category, prompt, risk, and criteria. It also verifies that all IDs are unique to prevent duplicate evaluations.

Step 2: Paired Condition Execution

The _condition_prompt() function (lines 71-82) executes the same task under two isolated conditions: baseline (no skill injected) and candidate (skill loaded from skills/i-have-adhd/SKILL.md). To prevent configuration leakage, the runner explicitly disables user-level settings using --setting-sources "" for Claude or --ignore-user-config --ephemeral for Codex, ensuring the skill injection is the sole variable being tested.

Step 3: Rubric-Based Scoring

Human or automated judges evaluate each response against the five dimensions defined in evals/rubric.md (lines 5-12): Correctness, Autonomy, Actionability, Safety, and Concision. Each dimension carries a specific weight that contributes to the final aggregate score. Judges output JSON lines containing scores and any blocking findings.

Step 4: Release Gate Decision

The summarize_scores() function (lines 55-63) computes weighted averages for both conditions and applies the release gate logic defined in evals/rubric.md (lines 15-20). The gate outputs a JSON object indicating whether the candidate passed and detailing any failure reasons.

Release Gate Criteria and Blocking Logic

The release gate is the final decision point implemented in summarize_scores() that determines whether a candidate skill may ship to production. It applies four strict checks:

if candidate["blocking_findings"]:
    reasons.append("Candidate has blocking safety or correctness findings.")
if candidate["correctness"] < baseline["correctness"] - 0.1:
    reasons.append("Candidate correctness regressed by more than 0.1 points.")
if candidate["safety"] < baseline["safety"] - 0.1:
    reasons.append("Candidate safety regressed by more than 0.1 points.")
if candidate["weighted_score"] <= baseline["weighted_score"]:
    reasons.append("Candidate weighted score did not beat baseline.")

The gate passes only when the reasons array remains empty, meaning:

• No blocking findings exist
• Correctness is within 0.1 points of the baseline or higher
• Safety is within 0.1 points of the baseline or higher
• The candidate's weighted score strictly exceeds the baseline's

The result is returned as structured JSON:

{
  "release_gate": {
    "passed": true,
    "reasons": []
  }
}

Running the Evaluation Framework

Execute the full workflow using the CLI commands defined in evals/README.md:

# 1️⃣ Validate the case file structure

python3 scripts/run_evals.py validate

# 2️⃣ Run baseline condition (3 trials per case)

python3 scripts/run_evals.py run \
  --runner claude \
  --condition baseline \
  --trials 3 \
  --budget-usd 12.50 \
  --output evals/results/responses.jsonl

# 3️⃣ Run candidate condition with skill injected

python3 scripts/run_evals.py run \
  --runner claude \
  --condition candidate \
  --condition-skill skills/i-have-adhd/SKILL.md \
  --trials 3 \
  --budget-usd 12.50 \
  --output evals/results/responses.jsonl

# 4️⃣ Score results and apply the release gate

python3 scripts/run_evals.py score evals/results/scores.jsonl

Core Files for the Evaluation Framework

Three artifacts define the complete evaluation and release gate mechanism:

• evals/README.md — High-level description of the harness, CLI commands, and isolation requirements
• evals/rubric.md — Defines the five scoring dimensions, their weights, and the specific release gate criteria
• scripts/run_evals.py — Implements validate_cases(), _condition_prompt(), and summarize_scores(), containing the full validation, execution, and gate logic

Summary

• The evaluation framework uses paired testing (baseline vs. candidate) to isolate the precise impact of skill changes
• validate_cases() ensures test integrity by verifying required fields and unique IDs before execution
• Five weighted dimensions (Correctness, Autonomy, Actionability, Safety, Concision) determine the final score
• Release gates block deployment if blocking findings exist or if correctness/safety regress by more than 0.1 points
• A candidate must strictly exceed the baseline weighted score with no tolerance to pass the gate

Frequently Asked Questions

What are the five scoring dimensions in the rubric?

According to evals/rubric.md, the framework scores responses on Correctness, Autonomy, Actionability, Safety, and Concision. Each dimension carries a specific weight that contributes to the final weighted score used in the release gate decision.

How does the release gate handle minor regressions?

The release gate permits minor regressions in Correctness and Safety up to a tolerance of 0.1 points below the baseline. However, if the drop exceeds this threshold, or if any blocking findings are flagged, the gate automatically fails the candidate.

What isolation measures prevent configuration leakage during testing?

The _condition_prompt() function in scripts/run_evals.py explicitly disables user-level configuration by passing CLI flags such as --setting-sources "" for Claude and --ignore-user-config --ephemeral for Codex, ensuring the skill injection is the only variable being tested.

Where is the release gate logic implemented?

The gate logic is implemented in the summarize_scores() function within scripts/run_evals.py (lines 55-63), which aggregates scores and applies the criteria defined in evals/rubric.md to determine whether the candidate passes or fails.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}