# How to Interpret Results from the i-have-adhd Skill Evaluations: A Step-by-Step Guide

> Learn to interpret i-have-adhd skill evaluations. This guide breaks down the JSON-Lines results, weighted scores, and quality thresholds for accurate analysis.

- Repository: [Ayoub Ghriss/i-have-adhd](https://github.com/ayghri/i-have-adhd)
- Tags: how-to-guide
- Published: 2026-08-03

---

**The i-have-adhd skill evaluator produces a JSON-Lines file with per-trial records and a summary object that aggregates weighted scores and enforces release-gate rules to determine if the skill meets quality thresholds.**

This guide walks you through interpreting skill evaluation results from the `ayghri/i-have-adhd` open-source repository. You'll learn to read the output artifacts, understand the scoring logic, and diagnose common failures using the actual implementation in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py).

---

## Understanding the Evaluation Output Files

Running the evaluator generates two primary artifacts: a **JSON-Lines file** (`.jsonl`) containing raw trial data, and a **summary dictionary** with aggregated metrics and pass/fail status.

### The JSON-Lines Record Structure

Each line in the output file represents one trial evaluation. The schema is defined in the `_score_row` helper within [`tests/test_run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/tests/test_run_evals.py) and contains these fields:

| Field | Type | Description |
|-------|------|-------------|
| `case_id` | `str` | Unique task identifier (e.g., `task_001`). |
| `condition` | `str` | Rubric dimension being scored: `clarity`, `actionability`, or `conciseness`. |
| `trial` | `int` | Trial number starting at 1 (default: 3 trials per case). |
| `score` | `int` | Model-assigned rating from 0 to 100. |
| `explanation` | `str` | Optional justification text from the LLM. |
| `confidence` | `float` | Optional model confidence score (may be `null`). |

```json
{
  "case_id": "task_001",
  "condition": "actionability",
  "trial": 2,
  "score": 92,
  "explanation": "The answer lists concrete next steps with clear ordering.",
  "confidence": 0.87
}

```

The evaluator writes these records via `scripts/run_evals.py::run_evaluations` and validates them through `_parse_response` and `_validate_score`, which enforce that scores are integers in the 0-100 range.

---

## Decoding the Summary Object

After all trials complete, `summarize_scores` in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) produces a summary dictionary. This is the primary artifact for determining whether your skill passes quality gates.

### Key Summary Fields

| Field | Purpose |
|-------|---------|
| `cases` | Per-case aggregates: raw `scores`, arithmetic `average`, and `weighted_score` from rubric weighting. |
| `overall_average` | Mean of all case averages across the evaluation suite. |
| `release_gate_passed` | Boolean indicating whether every case met the minimum threshold. |
| `failed_cases` | List of `case_id` values that fell below the gate threshold. |
| `duplicate_rows` | Boolean confirming uniqueness of `(case_id, trial, condition)` tuples. |

```json
{
  "overall_average": 84.3,
  "release_gate_passed": true,
  "failed_cases": [],
  "cases": {
    "task_001":
      "average": 91,
      "weighted_score": 93.2,
      "scores": [92, 90, 91]
    },
    "task_002": {
      "average": 78,
      "weighted_score": 77.5,
      "scores": [80, 77, 77]
    }
  }
}

```

**Critical rule:** If `release_gate_passed` is `false`, the evaluator exits with a non-zero status code, blocking release.

---

## How Scoring and Weighting Work

The evaluation applies a **rubric-based weighting system** defined in [`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md). Each condition carries a weight that multiplies its average score to produce the case's final weighted score.

### Default Rubric Weights

- **Clarity**: 0.4
- **Actionability**: 0.3
- **Conciseness**: 0.3

The weighted score calculation happens in `summarize_scores`: `(clarity_avg × 0.4) + (actionability_avg × 0.3) + (conciseness_avg × 0.3)`.

### Release Gate Threshold

The default **minimum gate is 70**. Every case must achieve a `weighted_score` ≥ 70 for `release_gate_passed` to be true. Override this with the `--gate <value>` CLI argument parsed in `run_evaluations`.

---

## Running Evaluations and Inspecting Results

### Command-Line Execution

```bash

# Basic run with default settings (3 trials, gate=70)

python -m scripts.run_evals \
    --cases evals/cases.jsonl \
    --runner-config evals/runners.example.json

# Custom trial count and gate threshold

python -m scripts.run_evals \
    --cases evals/cases.jsonl \
    --runner-config evals/runners.example.json \
    --trials 5 \
    --gate 75

```

The command writes `run_results.jsonl` to the working directory and prints the JSON summary to **stdout**.

### Programmatic Result Analysis

```python
import jsonlines
from collections import defaultdict
import json
import subprocess

# Load and inspect raw results

def load_results(path="run_results.jsonl"):
    with jsonlines.open(path) as reader:
        return list(reader)

records = load_results()

# Aggregate scores by case

by_case = defaultdict(list)
for rec in records:
    by_case[rec["case_id"]].append(rec["score"])

# Calculate per-case statistics

for case_id, scores in by_case.items():
    avg = sum(scores) / len(scores)
    print(f"{case_id}: {avg:.1f} (n={len(scores)})")

```

### Automated Gate Checking

```python
import subprocess
import json
import sys

result = subprocess.run(
    ["python", "-m", "scripts.run_evals", "--gate", "70"],
    capture_output=True,
    text=True
)

summary = json.loads(result.stdout)

if not summary["release_gate_passed"]:
    print(f"Failed cases: {summary['failed_cases']}")
    sys.exit(1)

print(f"✅ Passed: {summary['overall_average']:.1f} overall average")

```

---

## Diagnosing Common Evaluation Failures

| Error / Symptom | Root Cause | Verification Method |
|-----------------|------------|---------------------|
| `DuplicateScoreRowsError` | Duplicate `(case_id, trial, condition)` tuples in output. | Check `_check_pairing` in test output; inspect `.jsonl` for duplicate lines. |
| Missing trial errors | Fewer trials than requested due to runtime abort. | Count occurrences per `case_id` in output; should equal `--trials` value. |
| `release_gate_passed: false` | One or more cases below threshold. | Review `failed_cases` list; re-run with `--cases` targeting specific failures. |
| `ValueError` from `_validate_score` | LLM returned non-integer or out-of-range score. | Examine test output for offending record; check model prompt compliance. |

The `scripts/run_evals.py::_check_pairing` function enforces data integrity by verifying each trial-condition combination appears exactly once. Failures here indicate race conditions or retry logic bugs in the runner configuration.

---

## Core Implementation Files

Reference these files directly to understand evaluation behavior:

- **[`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py)** — Main driver with `load_cases`, `validate_cases`, `run_evaluations`, `_parse_response`, `_validate_score`, `_check_pairing`, and `summarize_scores`.
- **[`tests/test_run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/tests/test_run_evals.py)** — Test-driven specification of evaluation contracts and edge cases.
- **[`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md)** — Weight definitions and release-gate documentation.
- **`evals/cases.jsonl`** — Task definitions evaluated by the harness.

---

## Summary

- **i-have-adhd skill evaluations** produce machine-readable artifacts: a `.jsonl` file with per-trial records and a JSON summary with aggregated metrics.
- **Interpret results** by checking `release_gate_passed` first, then drilling into `failed_cases` and per-case `weighted_score` values.
- **Weighted scoring** applies rubric weights from [`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md) to condition averages; the default 70-point gate must be cleared by every case.
- **Debug failures** using duplicate-row detection in `_check_pairing`, score validation in `_validate_score`, and targeted re-runs via CLI filters.

---

## Frequently Asked Questions

### What does the `release_gate_passed` field indicate?

The `release_gate_passed` boolean in the summary object signals whether all evaluated cases met the minimum quality threshold. When `false`, the `failed_cases` list identifies which `case_id` values fell below the weighted score gate, and the evaluator exits with a non-zero status code to block automated releases.

### How do I change the passing threshold for evaluations?

Pass the `--gate` argument to the CLI invocation. The `run_evaluations` function in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) parses this value and passes it to `summarize_scores`, which compares each case's `weighted_score` against your custom threshold instead of the default 70.

### Why am I seeing `DuplicateScoreRowsError` during evaluation?

This error from `_check_pairing` indicates that multiple rows share the same `(case_id, trial, condition)` tuple in the output file. Verify your runner configuration isn't producing duplicate API calls, and inspect the `.jsonl` file directly to identify colliding records.

### Can I run evaluation on a subset of cases?

Yes. Use the `--cases` argument to provide a custom path to your case file. The `load_cases` function in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) reads any valid JSON-Lines file with `case_id`, `task`, and `condition` fields, allowing targeted debugging of specific failure modes.