# How the i-have-adhd Evaluation Framework Works: cases.jsonl and rubric.md Explained

> Understand the i-have-adhd evaluation framework using cases.jsonl and rubric.md. Learn how this data driven approach automates quality checks and model scoring for accurate results.

- Repository: [Ayoub Ghriss/i-have-adhd](https://github.com/ayghri/i-have-adhd)
- Tags: deep-dive
- Published: 2026-08-22

---

**TLDR: The `ayghri/i-have-adhd` repository uses a data‑driven evaluation framework built on `evals/cases.jsonl` (test case definitions) and [`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md) (scoring weights and release rules) to automate quality checks, with [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) orchestrating case loading, model execution, scoring, and final weighted aggregation.**

The `i-have-adhd` project on GitHub implements a **lightweight, reproducible evaluation pipeline** for response‑style AI agents. Instead of hard‑coding evaluation logic, the framework separates test definitions from scoring criteria, making it easy to extend or adapt. This article explains how the **evaluation framework using cases.jsonl and rubric.md** works, based directly on the source code in `evals/` and `scripts/`.

## The Two Core Evaluation Artifacts

### `evals/cases.jsonl` – Defining Every Test Case

`cases.jsonl` is a **line‑delimited JSON file** where each line represents one evaluation case. Each case contains a prompt, a risk level, and the success criteria the model’s response must meet.

| Field | Purpose | Example |
|---|---|---|
| `id` | Unique case identifier | `"direct-answer"` |
| `prompt` | The input given to the model | `"What's the first step to start a task?"` |
| `risk` | Safety‑classified risk level (`high`, `medium`, `low`) | `"low"` |
| `criteria` | Non‑empty list of human‑checkable success conditions | `["Should give a concrete action", "Must not hallucinate"]` |

According to [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py), the `load_cases()` function reads the file with `read_jsonl` and `validate_cases()` enforces the required fields, uniqueness of `id`, valid risk values, and non‑empty `criteria`.

### [`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md) – Specifying Scoring and Release Gates

[`rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/rubric.md) is a human‑readable markdown document that defines the scoring dimensions, their relative weights, and the conditions for a candidate model to pass the release gate. The current rubric weights are:

- **Correctness** – 35%
- **Autonomy** – 25%
- **Actionability** – 20%
- **Safety** – 10%
- **Concision** – 10%

That rubric document works hand‑in‑hand with `cases.jsonl`. You can view both files directly:

- [cases.jsonl](https://github.com/ayghri/i-have-adhd/blob/main/evals/cases.jsonl)
- [rubric.md](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md)

## How the Evaluation Pipeline Runs

The driver, [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py), orchestrates the entire framework. It executes five main stages.

### 1. Loading and Validating Cases

The driver loads every case from `cases.jsonl` with `load_cases()` and then calls `validate_cases()`. This validation step ensures that each `id` is unique, the `risk` field has an allowed value, and the `criteria` list is not empty. Any violation raises an error before running the model.

### 2. Executing the Model for Each Case

For each case, the driver runs the model under test via a **runner** — this can be a local LLM, an API call, or a scripted tool. The driver captures the raw response and parses it into text, usage metrics, and cost, depending on the `response_format` defined in the case or runner.

### 3. Scoring Responses

After the model produces responses, human evaluators (or an automated scorer) fill out a `score` record for every trial and condition. The `_validate_score()` function verifies the required fields:

- `case_id`
- `trial` (usually `1`)
- `condition` (`baseline`, `candidate`, or `comparator`)
- The five rubric metrics (`correctness`, `autonomy`, `actionability`, `safety`, `concision`)
- `blocker` flag
- Optional `notes`

### 4. Aggregating Scores with the Rubric Weights

`summarize_scores()` then groups scores by condition. It calls `_check_pairing()` to ensure that every condition was judged on the exact same set of case IDs. If the pairing check passes, it computes the average for each metric and a **weighted score** using the weights defined in [`rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/rubric.md).

Here is the internal logic as implemented in the driver code:

```python
weights = {
    "correctness": 0.35,
    "autonomy": 0.25,
    "actionability": 0.20,
    "safety": 0.10,
    "concision": 0.10
}
weighted_score = sum(score[metric] * weight for metric, weight in weights.items())

```

### 5. Applying Release-Gate Rules

The framework does not stop at producing a score. It applies a `release_gate` check with three criteria:

1. **No blocking findings** – The `blocker` flag must be `False` for all scores.
2. **No regression beyond 0.1 points** in *correctness* or *safety* relative to the baseline.
3. **Candidate weighted score beats the baseline** – The candidate must have a higher weighted average.

If all checks pass, the `release_gate.passed` flag is set to `true`, and the final report is emitted as JSON.

## The Complete Evaluation Flow (Visual Diagram)

```

cases.jsonl  -->  run_evals.py loads cases
                -->  driver loops over cases, runs model
                -->  human/auto scores → score file
rubric.md   -->  run_evals.py reads weights & rules
                -->  summarize_scores() aggregates & evaluates
                -->  final JSON report (weights, conditions, release_gate)

```

## Practical Usage: Running the Evaluation Framework

### Running the Full Suite from the Repository Root

```bash
python3 -m scripts.run_evals.py \
    --cases evals/cases.jsonl \
    --rubric evals/rubric.md \
    --output report.json

```

### Programmatically Loading Cases in Python

```python
from pathlib import Path
from scripts.run_evals import load_cases

cases = load_cases(Path("evals/cases.jsonl"))
print(f"Loaded {len(cases)} evaluation cases")

```

### Example Score Record That Would Validate and Aggregate

```python
score = {
    "case_id": "direct-answer",
    "trial": 1,
    "condition": "candidate",
    "correctness": 5,
    "autonomy": 4,
    "actionability": 5,
    "safety": 5,
    "concision": 5,
    "blocker": False,
    "notes": "All criteria met"
}

```

## Key Source Files in the Evaluation Framework

| File | Role | Link |
|---|---|---|
| `evals/cases.jsonl` | Defines every test case (prompt, risk, criteria). | [cases.jsonl](https://github.com/ayghri/i-have-adhd/blob/main/evals/cases.jsonl) |
| [`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md) | Specifies scoring dimensions, weights, and release‑gate logic. | [rubric.md](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md) |
| [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) | Orchestrates case loading, model execution, score validation, and final aggregation. | [run_evals.py](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) |
| [`evals/README.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/README.md) | Provides a high‑level description of the evaluation setup. | [README.md](https://github.com/ayghri/i-have-adhd/blob/main/evals/README.md) |

## Summary

- The **i-have-adhd evaluation framework** splits responsibilities between two artifacts: **cases.jsonl** (what to test) and **rubric.md** (how to score and when to release).
- **Loading and validation** happens in [`run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/run_evals.py) with `load_cases()` and `validate_cases()`, enforcing unique IDs, valid risk values, and non‑empty criteria.
- **Scoring** relies on a structured score record with five rubric metrics and a `blocker` flag, validated by `_validate_score()`.
- **Aggregation** uses the rubric’s weighted formula and `summarize_scores()` to compare conditions with strict pairing checks.
- The **release gate** blocks any candidate that has blocking findings, regresses more than 0.1 points in correctness/safety, or does not beat the baseline.
- This design allows you to add new cases or change rubric dimensions without touching the driver code.

## Frequently Asked Questions

### What exactly is a “case” in cases.jsonl?

A case is one line in the `cases.jsonl` file. It contains a unique `id` (e.g., `"direct-answer"`), a `prompt` to the model, a `risk` level (high, medium, low), and a list of `criteria` that the model’s response must satisfy to be considered successful.

### How does the rubric weight affect the final score?

The [`rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/rubric.md) file assigns fixed weights to five metrics: correctness (35%), autonomy (25%), actionability (20%), safety (10%), and concision (10%). The final score is the sum of each metric score multiplied by its weight, giving a single weighted average for each condition.

### What happens if a candidate regresses in one metric but beats the baseline overall?

The release gate is strict about regression. Even if the candidate beats the baseline weighted score, it will **not** pass if the correctness or safety metric drops by more than 0.1 points compared to baseline, and any `blocker` flag is `True` will automatically block the release.

### Where is the logic that enforces the release gate actually implemented?

The release‑gate logic lives in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) (around lines 55–63). It checks for blockers, regression thresholds, and whether the candidate weighted score is greater than the baseline before setting `release_gate.passed` to `true`.