How to Interpret Results from the i-have-adhd Skill Evaluations: A Step-by-Step Guide

The i-have-adhd skill evaluator produces a JSON-Lines file with per-trial records and a summary object that aggregates weighted scores and enforces release-gate rules to determine if the skill meets quality thresholds.

This guide walks you through interpreting skill evaluation results from the ayghri/i-have-adhd open-source repository. You'll learn to read the output artifacts, understand the scoring logic, and diagnose common failures using the actual implementation in scripts/run_evals.py.

---

Understanding the Evaluation Output Files

Running the evaluator generates two primary artifacts: a JSON-Lines file (.jsonl) containing raw trial data, and a summary dictionary with aggregated metrics and pass/fail status.

The JSON-Lines Record Structure

Each line in the output file represents one trial evaluation. The schema is defined in the _score_row helper within tests/test_run_evals.py and contains these fields:

Field	Type	Description
case_id	str	Unique task identifier (e.g., task_001).
condition	str	Rubric dimension being scored: clarity, actionability, or conciseness.
trial	int	Trial number starting at 1 (default: 3 trials per case).
score	int	Model-assigned rating from 0 to 100.
explanation	str	Optional justification text from the LLM.
confidence	float	Optional model confidence score (may be null).

{
  "case_id": "task_001",
  "condition": "actionability",
  "trial": 2,
  "score": 92,
  "explanation": "The answer lists concrete next steps with clear ordering.",
  "confidence": 0.87
}

The evaluator writes these records via scripts/run_evals.py::run_evaluations and validates them through _parse_response and _validate_score, which enforce that scores are integers in the 0-100 range.

---

Decoding the Summary Object

After all trials complete, summarize_scores in scripts/run_evals.py produces a summary dictionary. This is the primary artifact for determining whether your skill passes quality gates.

Key Summary Fields

Field	Purpose
cases	Per-case aggregates: raw scores, arithmetic average, and weighted_score from rubric weighting.
overall_average	Mean of all case averages across the evaluation suite.
release_gate_passed	Boolean indicating whether every case met the minimum threshold.
failed_cases	List of case_id values that fell below the gate threshold.
duplicate_rows	Boolean confirming uniqueness of (case_id, trial, condition) tuples.

{
  "overall_average": 84.3,
  "release_gate_passed": true,
  "failed_cases": [],
  "cases": {
    "task_001":
      "average": 91,
      "weighted_score": 93.2,
      "scores": [92, 90, 91]
    },
    "task_002": {
      "average": 78,
      "weighted_score": 77.5,
      "scores": [80, 77, 77]
    }
  }
}

Critical rule: If release_gate_passed is false, the evaluator exits with a non-zero status code, blocking release.

---

How Scoring and Weighting Work

The evaluation applies a rubric-based weighting system defined in evals/rubric.md. Each condition carries a weight that multiplies its average score to produce the case's final weighted score.

Default Rubric Weights

• Clarity: 0.4
• Actionability: 0.3
• Conciseness: 0.3

The weighted score calculation happens in summarize_scores: (clarity_avg × 0.4) + (actionability_avg × 0.3) + (conciseness_avg × 0.3).

Release Gate Threshold

The default minimum gate is 70. Every case must achieve a weighted_score ≥ 70 for release_gate_passed to be true. Override this with the --gate <value> CLI argument parsed in run_evaluations.

---

Running Evaluations and Inspecting Results

Command-Line Execution

# Basic run with default settings (3 trials, gate=70)

python -m scripts.run_evals \
    --cases evals/cases.jsonl \
    --runner-config evals/runners.example.json

# Custom trial count and gate threshold

python -m scripts.run_evals \
    --cases evals/cases.jsonl \
    --runner-config evals/runners.example.json \
    --trials 5 \
    --gate 75

The command writes run_results.jsonl to the working directory and prints the JSON summary to stdout.

Programmatic Result Analysis

import jsonlines
from collections import defaultdict
import json
import subprocess

# Load and inspect raw results

def load_results(path="run_results.jsonl"):
    with jsonlines.open(path) as reader:
        return list(reader)

records = load_results()

# Aggregate scores by case

by_case = defaultdict(list)
for rec in records:
    by_case[rec["case_id"]].append(rec["score"])

# Calculate per-case statistics

for case_id, scores in by_case.items():
    avg = sum(scores) / len(scores)
    print(f"{case_id}: {avg:.1f} (n={len(scores)})")

Automated Gate Checking

import subprocess
import json
import sys

result = subprocess.run(
    ["python", "-m", "scripts.run_evals", "--gate", "70"],
    capture_output=True,
    text=True
)

summary = json.loads(result.stdout)

if not summary["release_gate_passed"]:
    print(f"Failed cases: {summary['failed_cases']}")
    sys.exit(1)

print(f"✅ Passed: {summary['overall_average']:.1f} overall average")

---

Diagnosing Common Evaluation Failures

Error / Symptom	Root Cause	Verification Method
DuplicateScoreRowsError	Duplicate (case_id, trial, condition) tuples in output.	Check _check_pairing in test output; inspect .jsonl for duplicate lines.
Missing trial errors	Fewer trials than requested due to runtime abort.	Count occurrences per case_id in output; should equal --trials value.
release_gate_passed: false	One or more cases below threshold.	Review failed_cases list; re-run with --cases targeting specific failures.
ValueError from _validate_score	LLM returned non-integer or out-of-range score.	Examine test output for offending record; check model prompt compliance.

The scripts/run_evals.py::_check_pairing function enforces data integrity by verifying each trial-condition combination appears exactly once. Failures here indicate race conditions or retry logic bugs in the runner configuration.

---

Core Implementation Files

Reference these files directly to understand evaluation behavior:

• scripts/run_evals.py — Main driver with load_cases, validate_cases, run_evaluations, _parse_response, _validate_score, _check_pairing, and summarize_scores.
• tests/test_run_evals.py — Test-driven specification of evaluation contracts and edge cases.
• evals/rubric.md — Weight definitions and release-gate documentation.
• evals/cases.jsonl — Task definitions evaluated by the harness.

---

Summary

• i-have-adhd skill evaluations produce machine-readable artifacts: a .jsonl file with per-trial records and a JSON summary with aggregated metrics.
• Interpret results by checking release_gate_passed first, then drilling into failed_cases and per-case weighted_score values.
• Weighted scoring applies rubric weights from evals/rubric.md to condition averages; the default 70-point gate must be cleared by every case.
• Debug failures using duplicate-row detection in _check_pairing, score validation in _validate_score, and targeted re-runs via CLI filters.

---

Frequently Asked Questions

What does the release_gate_passed field indicate?

The release_gate_passed boolean in the summary object signals whether all evaluated cases met the minimum quality threshold. When false, the failed_cases list identifies which case_id values fell below the weighted score gate, and the evaluator exits with a non-zero status code to block automated releases.

How do I change the passing threshold for evaluations?

Pass the --gate argument to the CLI invocation. The run_evaluations function in scripts/run_evals.py parses this value and passes it to summarize_scores, which compares each case's weighted_score against your custom threshold instead of the default 70.

Why am I seeing DuplicateScoreRowsError during evaluation?

This error from _check_pairing indicates that multiple rows share the same (case_id, trial, condition) tuple in the output file. Verify your runner configuration isn't producing duplicate API calls, and inspect the .jsonl file directly to identify colliding records.

Can I run evaluation on a subset of cases?

Yes. Use the --cases argument to provide a custom path to your case file. The load_cases function in scripts/run_evals.py reads any valid JSON-Lines file with case_id, task, and condition fields, allowing targeted debugging of specific failure modes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}