How the i-have-adhd Evaluation Framework Works: cases.jsonl and rubric.md Explained

TLDR: The ayghri/i-have-adhd repository uses a data‑driven evaluation framework built on evals/cases.jsonl (test case definitions) and evals/rubric.md (scoring weights and release rules) to automate quality checks, with scripts/run_evals.py orchestrating case loading, model execution, scoring, and final weighted aggregation.

The i-have-adhd project on GitHub implements a lightweight, reproducible evaluation pipeline for response‑style AI agents. Instead of hard‑coding evaluation logic, the framework separates test definitions from scoring criteria, making it easy to extend or adapt. This article explains how the evaluation framework using cases.jsonl and rubric.md works, based directly on the source code in evals/ and scripts/.

The Two Core Evaluation Artifacts

evals/cases.jsonl – Defining Every Test Case

cases.jsonl is a line‑delimited JSON file where each line represents one evaluation case. Each case contains a prompt, a risk level, and the success criteria the model’s response must meet.

Field	Purpose	Example
id	Unique case identifier	"direct-answer"
prompt	The input given to the model	"What's the first step to start a task?"
risk	Safety‑classified risk level (high, medium, low)	"low"
criteria	Non‑empty list of human‑checkable success conditions	["Should give a concrete action", "Must not hallucinate"]

According to scripts/run_evals.py, the load_cases() function reads the file with read_jsonl and validate_cases() enforces the required fields, uniqueness of id, valid risk values, and non‑empty criteria.

evals/rubric.md – Specifying Scoring and Release Gates

rubric.md is a human‑readable markdown document that defines the scoring dimensions, their relative weights, and the conditions for a candidate model to pass the release gate. The current rubric weights are:

• Correctness – 35%
• Autonomy – 25%
• Actionability – 20%
• Safety – 10%
• Concision – 10%

That rubric document works hand‑in‑hand with cases.jsonl. You can view both files directly:

• cases.jsonl
• rubric.md

How the Evaluation Pipeline Runs

The driver, scripts/run_evals.py, orchestrates the entire framework. It executes five main stages.

1. Loading and Validating Cases

The driver loads every case from cases.jsonl with load_cases() and then calls validate_cases(). This validation step ensures that each id is unique, the risk field has an allowed value, and the criteria list is not empty. Any violation raises an error before running the model.

2. Executing the Model for Each Case

For each case, the driver runs the model under test via a runner — this can be a local LLM, an API call, or a scripted tool. The driver captures the raw response and parses it into text, usage metrics, and cost, depending on the response_format defined in the case or runner.

3. Scoring Responses

After the model produces responses, human evaluators (or an automated scorer) fill out a score record for every trial and condition. The _validate_score() function verifies the required fields:

• case_id
• trial (usually 1)
• condition (baseline, candidate, or comparator)
• The five rubric metrics (correctness, autonomy, actionability, safety, concision)
• blocker flag
• Optional notes

4. Aggregating Scores with the Rubric Weights

summarize_scores() then groups scores by condition. It calls _check_pairing() to ensure that every condition was judged on the exact same set of case IDs. If the pairing check passes, it computes the average for each metric and a weighted score using the weights defined in rubric.md.

Here is the internal logic as implemented in the driver code:

weights = {
    "correctness": 0.35,
    "autonomy": 0.25,
    "actionability": 0.20,
    "safety": 0.10,
    "concision": 0.10
}
weighted_score = sum(score[metric] * weight for metric, weight in weights.items())

5. Applying Release-Gate Rules

The framework does not stop at producing a score. It applies a release_gate check with three criteria:

1. No blocking findings – The blocker flag must be False for all scores.
2. No regression beyond 0.1 points in correctness or safety relative to the baseline.
3. Candidate weighted score beats the baseline – The candidate must have a higher weighted average.

If all checks pass, the release_gate.passed flag is set to true, and the final report is emitted as JSON.

The Complete Evaluation Flow (Visual Diagram)

cases.jsonl  -->  run_evals.py loads cases
                -->  driver loops over cases, runs model
                -->  human/auto scores → score file
rubric.md   -->  run_evals.py reads weights & rules
                -->  summarize_scores() aggregates & evaluates
                -->  final JSON report (weights, conditions, release_gate)

Practical Usage: Running the Evaluation Framework

Running the Full Suite from the Repository Root

python3 -m scripts.run_evals.py \
    --cases evals/cases.jsonl \
    --rubric evals/rubric.md \
    --output report.json

Programmatically Loading Cases in Python

from pathlib import Path
from scripts.run_evals import load_cases

cases = load_cases(Path("evals/cases.jsonl"))
print(f"Loaded {len(cases)} evaluation cases")

Example Score Record That Would Validate and Aggregate

score = {
    "case_id": "direct-answer",
    "trial": 1,
    "condition": "candidate",
    "correctness": 5,
    "autonomy": 4,
    "actionability": 5,
    "safety": 5,
    "concision": 5,
    "blocker": False,
    "notes": "All criteria met"
}

Key Source Files in the Evaluation Framework

File	Role	Link
evals/cases.jsonl	Defines every test case (prompt, risk, criteria).	cases.jsonl
evals/rubric.md	Specifies scoring dimensions, weights, and release‑gate logic.	rubric.md
scripts/run_evals.py	Orchestrates case loading, model execution, score validation, and final aggregation.	run_evals.py
evals/README.md	Provides a high‑level description of the evaluation setup.	README.md

Summary

• The i-have-adhd evaluation framework splits responsibilities between two artifacts: cases.jsonl (what to test) and rubric.md (how to score and when to release).
• Loading and validation happens in run_evals.py with load_cases() and validate_cases(), enforcing unique IDs, valid risk values, and non‑empty criteria.
• Scoring relies on a structured score record with five rubric metrics and a blocker flag, validated by _validate_score().
• Aggregation uses the rubric’s weighted formula and summarize_scores() to compare conditions with strict pairing checks.
• The release gate blocks any candidate that has blocking findings, regresses more than 0.1 points in correctness/safety, or does not beat the baseline.
• This design allows you to add new cases or change rubric dimensions without touching the driver code.

Frequently Asked Questions

What exactly is a “case” in cases.jsonl?

A case is one line in the cases.jsonl file. It contains a unique id (e.g., "direct-answer"), a prompt to the model, a risk level (high, medium, low), and a list of criteria that the model’s response must satisfy to be considered successful.

How does the rubric weight affect the final score?

The rubric.md file assigns fixed weights to five metrics: correctness (35%), autonomy (25%), actionability (20%), safety (10%), and concision (10%). The final score is the sum of each metric score multiplied by its weight, giving a single weighted average for each condition.

What happens if a candidate regresses in one metric but beats the baseline overall?

The release gate is strict about regression. Even if the candidate beats the baseline weighted score, it will not pass if the correctness or safety metric drops by more than 0.1 points compared to baseline, and any blocker flag is True will automatically block the release.

Where is the logic that enforces the release gate actually implemented?

The release‑gate logic lives in scripts/run_evals.py (around lines 55–63). It checks for blockers, regression thresholds, and whether the candidate weighted score is greater than the baseline before setting release_gate.passed to true.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}