Key Models and Data Structures in the i-have-adhd Project

The i-have-adhd repository relies on immutable JSON-L case definitions, weighted score rows, and YAML-based skill manifests to execute paired LLM evaluations under different response-style conditions.

The i-have-adhd project is a lightweight evaluation harness designed to compare LLM response quality across distinct "response-style" configurations. Understanding the key models and data structures is essential for extending the framework or interpreting evaluation results. The architecture centers on plain Python dictionaries and configuration files that drive the evaluation pipeline from case definition to final aggregation.

Core Evaluation Data Structures

The evaluation engine in scripts/run_evals.py defines three primary data structures that govern how trials are executed and scored.

Evaluation Case

The Evaluation Case is a JSON-L object that describes a single test scenario. Each case must contain an id, category, prompt, risk level, and a list of criteria used for scoring. The harness validates these objects using the validate_cases() function before execution begins.

In scripts/run_evals.py, the validation logic ensures every case adheres to the expected schema:

from scripts.run_evals import load_cases, validate_cases

cases = load_cases()                     # reads evals/cases.jsonl

errors = validate_cases(cases)           # returns [] if every case is well‑formed

if errors:
    raise ValueError("\n".join(errors))

The case catalog serves as the driver for every evaluation run, with each case representing a unique scenario against which LLM responses are tested.

Score Row

A Score Row represents the result of a single trial for a given case and condition. According to the _validate_score() function in scripts/run_evals.py, each row contains:

• The case identifier and trial number
• The condition name (baseline, candidate, or comparator) and runner name
• The LLM's raw response and usage metrics
• A boolean blocker flag indicating critical failures
• Numeric ratings (1-5) for the five weighted metrics

These rows are appended to the output JSON-L file during execution and form the raw data for downstream analysis.

Weights and Conditions

The evaluation framework uses two critical constants defined at the module level in scripts/run_evals.py:

• WEIGHTS: A mapping that defines the relative importance of each metric (correctness, autonomy, actionability, safety, conciseness). These weights feed into the weighted aggregate score calculation.
• CONDITIONS: The set of allowed evaluation conditions (baseline, candidate, comparator) that determine which rows can be compared against each other during aggregation.

Skill Configuration Models

Beyond the evaluation harness, the project defines several metadata structures that register and configure the ADHD-friendly response skill.

Plugin Manifest

The plugin.json file contains minimal metadata that registers the skill with the host platform. This JSON object specifies the skill name and a short description, enabling the platform to discover and load the plugin correctly.

Skill Interface

Located at skills/i-have-adhd/agents/openai.yaml, the Skill Interface describes how the i-have-adhd skill is invoked. This YAML file defines the display name, short description, and default prompt presented to users in the host interface.

Skill Specification

The full behavioral specification resides in .cursor/skills/i-have-adhd/SKILL.md. This markdown file contains the complete set of output-shaping rules that dictate ADHD-friendly response characteristics. When running non-baseline conditions, the harness wraps this content in a <response_style> block to instruct the LLM on formatting constraints.

Working with the Data Structures

The evaluation pipeline follows a clear data flow from case ingestion to result aggregation.

Loading and Validating Cases

The harness reads the case catalog from evals/cases.jsonl using load_cases() and validates schema compliance via validate_cases() (lines 44-46 and 59-78 in scripts/run_evals.py).

Running Evaluations

The run_evaluations() function (lines 52-100) orchestrates the trial execution. For each case, it builds a command line based on the selected runner configuration from evals/runners.example.json. When a non-baseline condition is requested, the script injects the skill instructions from SKILL.md:

python -m scripts.run_evals run \
    --runner my-runner \
    --condition candidate \
    --condition-skill .cursor/skills/i-have-adhd/SKILL.md \
    --output evals/results.jsonl

This command loads each case, injects the response-style instructions, executes the LLM via the runner, and appends a Score Row to the output file.

Aggregating Results

After collecting score rows, the summarize_scores() function (lines 30-68) groups data by condition, averages each metric using the WEIGHTS table, and produces a final "release gate" decision:

from scripts.run_evals import summarize_scores, read_jsonl
from pathlib import Path

rows = read_jsonl(Path("evals/results.jsonl"))
summary = summarize_scores(rows)
print(summary["release_gate"]["passed"])   # True → candidate outperforms baseline

Summary

• Evaluation Cases are immutable JSON-L objects defined in evals/cases.jsonl and validated by validate_cases() in scripts/run_evals.py.
• Score Rows capture trial results including metric ratings, usage data, and blocker flags as defined in _validate_score().
• Weights and Conditions constants (WEIGHTS and CONDITIONS) in scripts/run_evals.py govern scoring calculations and valid comparison groups.
• Skill Configuration spans three files: plugin.json for registration, openai.yaml for interface definitions, and SKILL.md for behavioral specifications.
• The architecture uses plain dictionaries to maintain portability across different LLM providers and evaluation scenarios.

Frequently Asked Questions

What fields are required in an Evaluation Case object?

Each Evaluation Case must include an id, category, prompt, risk level, and a list of criteria. The validate_cases() function in scripts/run_evals.py enforces this schema and returns detailed error messages for malformed entries.

How does the scoring aggregation work?

The summarize_scores() function groups Score Rows by condition, calculates weighted averages using the WEIGHTS constant (correctness, autonomy, actionability, safety, conciseness), and compares candidate performance against baseline thresholds to determine a pass/fail release gate.

Where is the ADHD-friendly response behavior defined?

The behavioral rules are specified in .cursor/skills/i-have-adhd/SKILL.md. During evaluation runs, this markdown content is wrapped in a <response_style> block and prepended to the prompt when the condition is set to candidate or comparator.

Can I add new metrics to the evaluation framework?

Yes. You can extend the WEIGHTS dictionary in scripts/run_evals.py to include additional metrics, provided you update the Score Row validation logic in _validate_score() and ensure your runner scripts output the new metric fields in their JSON responses.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}