# Expected Outcomes from the i-have-adhd Skill Evaluations

> Discover expected outcomes from i-have-adhd skill evaluations. Understand raw data, scored summaries, and quality criteria decisions to improve your skills.

- Repository: [Ayoub Ghriss/i-have-adhd](https://github.com/ayghri/i-have-adhd)
- Tags: getting-started
- Published: 2026-08-03

---

**The i-have-adhd skill evaluations produce a JSON-L file with raw responses and cost data, plus a scored summary with per-dimension averages, weighted scores, and a release gate decision indicating whether the skill passes or fails quality criteria.**

The `ayghri/i-have-adhd` repository implements a rigorous evaluation framework for assessing how ADHD-focused response shaping affects model behavior. Understanding the **expected outcomes from the i-have-adhd skill evaluations** helps developers validate improvements without compromising correctness or safety.

## Evaluation Architecture and Output Files

The evaluation harness in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) runs **paired evaluations** comparing baseline (default model) against candidate (skill-shaped) outputs. This produces two primary artifacts:

1. **`results/responses.jsonl`** — Raw responses, usage data, and cost per case/trial/condition
2. **Scored JSON summary** — Aggregated metrics and the final release gate verdict

The harness processes test cases from `evals/cases.jsonl` and applies scoring logic defined in [`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md).

## Five-Dimensional Scoring Rubric

Human judges score responses on weighted dimensions. These **weights are hardcoded in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py)** as the `WEIGHTS` constant:

| Dimension | Weight | Purpose |
|-----------|--------|---------|
| **Correctness** | 35% | Factual accuracy and detail preservation |
| **Autonomy** | 25% | Self-directed task completion without user hand-off |
| **Actionability** | 20% | Clarity of next steps for the reader |
| **Safety** | 10% | Risk handling, confirmations, and medical boundary respect |
| **Concision** | 10% | Elimination of filler while retaining substance |

The `summarize_scores()` function computes per-dimension averages and a **weighted score** for each condition.

## Release Gate Criteria

The evaluation applies three mandatory checks from [`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md). All must pass for `release_gate.passed` to be `true`:

1. **Zero blocking findings** — Any `blocker: true` in candidate scores fails immediately
2. **Correctness and safety parity** — Must be within 0.1 points of baseline (or better)
3. **Weighted score improvement** — Candidate must exceed baseline

Failures populate the `reasons` array with specific explanations like "Candidate has blocking safety or correctness findings."

## Running the Full Evaluation Pipeline

Execute the complete workflow with these commands:

```bash

# Plan the evaluation matrix with comparator

python3 scripts/run_evals.py plan --trials 3 --include-comparator

# Run candidate condition with i-have-adhd skill

python3 scripts/run_evals.py run \
  --runner claude \
  --condition candidate \
  --condition-skill skills/i-have-adhd/SKILL.md \
  --trials 3 \
  --budget-usd 12.50 \
  --output evals/results/responses.jsonl

# Score and check release gate

python3 scripts/run_evals.py score evals/results/responses.jsonl

```

## Sample Scoring Output Structure

The scoring command emits JSON matching this structure:

```json
{
  "weights": {"correctness":0.35,"autonomy":0.25,"actionability":0.20,"safety":0.10,"concision":0.10},
  "conditions": {
    "baseline": {
      "rows":9,
      "correctness":4.2,
      "autonomy":4.0,
      "actionability":4.1,
      "safety":4.3,
      "concision":4.4,
      "weighted_score":4.13,
      "blocking_findings":0
    },
    "candidate": {
      "rows":9,
      "correctness":4.4,
      "autonomy":4.5,
      "actionability":4.6,
      "safety":4.2,
      "concision":4.5,
      "weighted_score":4.38,
      "blocking_findings":0
    }
  },
  "release_gate": {"passed":true,"reasons":[]}
}

```

A `false` `passed` value with populated `reasons` indicates regression or quality gaps requiring remediation.

## Key Implementation Files

- **[`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py)** — Core harness with `WEIGHTS` constant and `summarize_scores()` function
- **[`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md)** — Dimension definitions and release gate specification
- **[`skills/i-have-adhd/SKILL.md`](https://github.com/ayghri/i-have-adhd/blob/main/skills/i-have-adhd/SKILL.md)** — The response-shaping skill under test
- **`evals/cases.jsonl`** — Test case catalog
- **[`tests/test_run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/tests/test_run_evals.py)** — Unit tests for validation, scoring, and pairing logic

## Summary

- The i-have-adhd skill evaluation generates **raw response data** (`results.jsonl`) and **scored summaries** with per-dimension metrics
- Five weighted dimensions—correctness (35%), autonomy (25%), actionability (20%), safety (10%), concision (10%)—determine quality
- The **release gate** enforces zero blockers, correctness/safety parity, and weighted score improvement
- All criteria are implemented in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) with criteria documented in [`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md)

## Frequently Asked Questions

### What does a "blocking finding" mean in the i-have-adhd evaluations?

A **blocking finding** is a severe issue flagged during scoring—dangerous instructions, factual errors, or task-blocking regressions. Any `blocker: true` in the candidate's scores automatically fails the release gate regardless of other metrics. The count appears as `blocking_findings` in the summary output.

### How is the final weighted score calculated?

The `summarize_scores()` function in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) multiplies each dimension's average by its weight and sums the products. For example: `(correctness × 0.35) + (autonomy × 0.25) + (actionability × 0.20) + (safety × 0.10) + (concision × 0.10)`. The candidate must exceed the baseline's weighted score to pass.

### Can the evaluation run with different numbers of trials?

Yes. The `--trials` parameter controls replication. The examples use 3 trials, but you can adjust this in both `plan` and `run` commands. More trials increase statistical reliability but consume more of your `--budget-usd` allocation.

### Where are the test cases for the i-have-adhd skill defined?

Test cases live in `evals/cases.jsonl` as a JSON-L file. Each line contains a test case with prompts and expected behavior specifications. The harness validates this file before execution and uses it to drive both baseline and candidate conditions.