# Key Models and Data Structures in the i-have-adhd Project

> Explore key models and data structures in the i-have-adhd project including JSON-L case definitions, weighted score rows, and YAML skill manifests for LLM evaluations.

- Repository: [Ayoub Ghriss/i-have-adhd](https://github.com/ayghri/i-have-adhd)
- Tags: deep-dive
- Published: 2026-07-30

---

**The i-have-adhd repository relies on immutable JSON-L case definitions, weighted score rows, and YAML-based skill manifests to execute paired LLM evaluations under different response-style conditions.**

The *i-have-adhd* project is a lightweight evaluation harness designed to compare LLM response quality across distinct "response-style" configurations. Understanding the key models and data structures is essential for extending the framework or interpreting evaluation results. The architecture centers on plain Python dictionaries and configuration files that drive the evaluation pipeline from case definition to final aggregation.

## Core Evaluation Data Structures

The evaluation engine in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) defines three primary data structures that govern how trials are executed and scored.

### Evaluation Case

The **Evaluation Case** is a JSON-L object that describes a single test scenario. Each case must contain an `id`, `category`, `prompt`, `risk` level, and a list of `criteria` used for scoring. The harness validates these objects using the `validate_cases()` function before execution begins.

In [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py), the validation logic ensures every case adheres to the expected schema:

```python
from scripts.run_evals import load_cases, validate_cases

cases = load_cases()                     # reads evals/cases.jsonl

errors = validate_cases(cases)           # returns [] if every case is well‑formed

if errors:
    raise ValueError("\n".join(errors))

```

The case catalog serves as the driver for every evaluation run, with each case representing a unique scenario against which LLM responses are tested.

### Score Row

A **Score Row** represents the result of a single trial for a given case and condition. According to the `_validate_score()` function in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py), each row contains:

- The case identifier and trial number
- The condition name (`baseline`, `candidate`, or `comparator`) and runner name
- The LLM's raw response and usage metrics
- A boolean `blocker` flag indicating critical failures
- Numeric ratings (1-5) for the five weighted metrics

These rows are appended to the output JSON-L file during execution and form the raw data for downstream analysis.

### Weights and Conditions

The evaluation framework uses two critical constants defined at the module level in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py):

- **`WEIGHTS`**: A mapping that defines the relative importance of each metric (`correctness`, `autonomy`, `actionability`, `safety`, `conciseness`). These weights feed into the weighted aggregate score calculation.
- **`CONDITIONS`**: The set of allowed evaluation conditions (`baseline`, `candidate`, `comparator`) that determine which rows can be compared against each other during aggregation.

## Skill Configuration Models

Beyond the evaluation harness, the project defines several metadata structures that register and configure the ADHD-friendly response skill.

### Plugin Manifest

The [`plugin.json`](https://github.com/ayghri/i-have-adhd/blob/main/plugin.json) file contains minimal metadata that registers the skill with the host platform. This JSON object specifies the skill name and a short description, enabling the platform to discover and load the plugin correctly.

### Skill Interface

Located at [`skills/i-have-adhd/agents/openai.yaml`](https://github.com/ayghri/i-have-adhd/blob/main/skills/i-have-adhd/agents/openai.yaml), the **Skill Interface** describes how the *i-have-adhd* skill is invoked. This YAML file defines the display name, short description, and default prompt presented to users in the host interface.

### Skill Specification

The full behavioral specification resides in [`.cursor/skills/i-have-adhd/SKILL.md`](https://github.com/ayghri/i-have-adhd/blob/main/.cursor/skills/i-have-adhd/SKILL.md). This markdown file contains the complete set of output-shaping rules that dictate ADHD-friendly response characteristics. When running non-baseline conditions, the harness wraps this content in a `<response_style>` block to instruct the LLM on formatting constraints.

## Working with the Data Structures

The evaluation pipeline follows a clear data flow from case ingestion to result aggregation.

### Loading and Validating Cases

The harness reads the case catalog from `evals/cases.jsonl` using `load_cases()` and validates schema compliance via `validate_cases()` (lines 44-46 and 59-78 in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py)).

### Running Evaluations

The `run_evaluations()` function (lines 52-100) orchestrates the trial execution. For each case, it builds a command line based on the selected runner configuration from [`evals/runners.example.json`](https://github.com/ayghri/i-have-adhd/blob/main/evals/runners.example.json). When a non-baseline condition is requested, the script injects the skill instructions from [`SKILL.md`](https://github.com/ayghri/i-have-adhd/blob/main/SKILL.md):

```bash
python -m scripts.run_evals run \
    --runner my-runner \
    --condition candidate \
    --condition-skill .cursor/skills/i-have-adhd/SKILL.md \
    --output evals/results.jsonl

```

This command loads each case, injects the response-style instructions, executes the LLM via the runner, and appends a Score Row to the output file.

### Aggregating Results

After collecting score rows, the `summarize_scores()` function (lines 30-68) groups data by condition, averages each metric using the `WEIGHTS` table, and produces a final "release gate" decision:

```python
from scripts.run_evals import summarize_scores, read_jsonl
from pathlib import Path

rows = read_jsonl(Path("evals/results.jsonl"))
summary = summarize_scores(rows)
print(summary["release_gate"]["passed"])   # True → candidate outperforms baseline

```

## Summary

- **Evaluation Cases** are immutable JSON-L objects defined in `evals/cases.jsonl` and validated by `validate_cases()` in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py).
- **Score Rows** capture trial results including metric ratings, usage data, and blocker flags as defined in `_validate_score()`.
- **Weights and Conditions** constants (`WEIGHTS` and `CONDITIONS`) in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) govern scoring calculations and valid comparison groups.
- **Skill Configuration** spans three files: [`plugin.json`](https://github.com/ayghri/i-have-adhd/blob/main/plugin.json) for registration, [`openai.yaml`](https://github.com/ayghri/i-have-adhd/blob/main/openai.yaml) for interface definitions, and [`SKILL.md`](https://github.com/ayghri/i-have-adhd/blob/main/SKILL.md) for behavioral specifications.
- The architecture uses plain dictionaries to maintain portability across different LLM providers and evaluation scenarios.

## Frequently Asked Questions

### What fields are required in an Evaluation Case object?

Each Evaluation Case must include an `id`, `category`, `prompt`, `risk` level, and a list of `criteria`. The `validate_cases()` function in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) enforces this schema and returns detailed error messages for malformed entries.

### How does the scoring aggregation work?

The `summarize_scores()` function groups Score Rows by condition, calculates weighted averages using the `WEIGHTS` constant (correctness, autonomy, actionability, safety, conciseness), and compares candidate performance against baseline thresholds to determine a pass/fail release gate.

### Where is the ADHD-friendly response behavior defined?

The behavioral rules are specified in [`.cursor/skills/i-have-adhd/SKILL.md`](https://github.com/ayghri/i-have-adhd/blob/main/.cursor/skills/i-have-adhd/SKILL.md). During evaluation runs, this markdown content is wrapped in a `<response_style>` block and prepended to the prompt when the condition is set to `candidate` or `comparator`.

### Can I add new metrics to the evaluation framework?

Yes. You can extend the `WEIGHTS` dictionary in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) to include additional metrics, provided you update the Score Row validation logic in `_validate_score()` and ensure your runner scripts output the new metric fields in their JSON responses.