What Data Is Used for Evaluating the i‑Have‑ADHD Skill? A Deep Dive into the Evaluation Framework

The i‑have‑adHD skill is evaluated using a curated catalog of test cases stored in evals/cases.jsonl, paired with human‑produced score rows that measure five metrics across three risk tiers.

The ayghri/i‑have‑adhd repository implements a rigorous, data‑driven evaluation pipeline to assess skill performance before release. Understanding this evaluation data structure is essential for contributors who want to add test cases, interpret results, or modify the scoring logic.

The Evaluation Data: Two Core Components

The evaluation system relies on two distinct data sources that work together to validate skill behavior.

1. The Case Catalog (evals/cases.jsonl)

Each line in evals/cases.jsonl is a JSON object representing a single evaluation case with five required fields. The validate_cases() function in scripts/run_evals.py enforces this schema:

Field	Purpose	Constraints
id	Unique case identifier	Must be unique across all cases
category	Functional grouping	Examples: direct‑answer, coding, safety
prompt	User query sent to the skill	Arbitrary text string
risk	Risk classification	Must be low, medium, or high
criteria	Success rubric items	Non‑empty list of requirement strings

The validation logic in scripts/run_evals.py (lines 59–79) rejects any case missing fields, with duplicate IDs, or with invalid risk levels. High‑risk cases receive stricter judgment criteria during human evaluation.

2. Human‑Produced Score Rows

After running a case through the skill, human judges produce a score row with quantitative assessments. According to the source code in scripts/run_evals.py (lines 82–95), each score row must contain:

• case_id — links back to the case catalog
• trial and condition — tracks experimental runs (baseline vs. candidate)
• Five metric scores: correctness, autonomy, actionability, safety, concision
• blocker — boolean flag for critical failures
• notes — free‑form qualitative feedback

Loading and Validating Evaluation Cases

The run_evals.py script provides load_cases() and validate_cases() functions to handle the case catalog. Here's how to use them programmatically:

from pathlib import Path
from scripts.run_evals import load_cases, validate_cases

cases_path = Path("evals/cases.jsonl")

# Load the case catalog

cases = load_cases(cases_path)  # internally calls read_jsonl()

# Validate schema compliance

errors = validate_cases(cases)
if errors:
    raise ValueError(f"Validation failed:\n" + "\n".join(errors))

print(f"Loaded {len(cases)} valid evaluation cases")

The read_jsonl() utility (lines 44–45) handles line‑delimited JSON parsing, returning a list of dictionaries for downstream processing.

Running Evaluations and Aggregating Scores

The evaluation pipeline executes cases through a configured LLM runner and collects human judgments. Use the CLI to execute a full evaluation run:

python -m scripts.run_evals run \
  --runner stub \
  --condition candidate \
  --output results/candidate.jsonl \
  --budget-usd 5.0 \
  --allow-unmetered

After scoring, aggregate results with summarize_scores():

from pathlib import Path
from scripts.run_evals import read_jsonl, summarize_scores

scores_path = Path("results/scores.jsonl")
scores = read_jsonl(scores_path)

summary = summarize_scores(scores)

# Access condition‑level results

baseline = summary["conditions"]["baseline"]["weighted_score"]
candidate = summary["conditions"]["candidate"]["weighted_score"]

print(f"Baseline: {baseline:.3f} | Candidate: {candidate:.3f}")
print(f"Release gate passed: {summary['release_gate']['passed']}")

The Release Gate Decision Logic

The summarize_scores() function in scripts/run_evals.py (lines 30–68) implements a weighted scoring algorithm that determines release readiness. The process:

1. Applies metric weights — defined in the WEIGHTS map within run_evals.py
2. Computes per‑condition averages — aggregates across all cases and trials
3. Evaluates three release criteria:
   • No blocker flags in any scored case
   • No regression exceeding 0.1 points from baseline
   • Candidate weighted score strictly greater than baseline

Failure on any criterion prevents automatic release approval.

Key Files in the Evaluation System

File	Role
evals/cases.jsonl	Master catalog of evaluation prompts, risk tiers, and success criteria
scripts/run_evals.py	Core engine: validation, LLM invocation, response parsing, score aggregation
tests/test_run_evals.py	Unit tests for validation logic, summarization, and budget enforcement
evals/runners.example.json	Example configuration for LLM runner backends

Summary

• Evaluation data for the i‑have‑adHD skill splits into structured case definitions (evals/cases.jsonl) and quantitative score rows from human judges.
• Five required fields per case (id, category, prompt, risk, criteria) are strictly validated by validate_cases() in scripts/run_evals.py.
• Five metrics (correctness, autonomy, actionability, safety, concision) plus blocker flags feed into a weighted release gate that compares candidate against baseline performance.
• The run_evals.py module serves as the central orchestrator for loading, validating, executing, and summarizing all evaluation data.

Frequently Asked Questions

How do I add a new evaluation case to the i‑have‑adHD skill?

Append a JSON object to evals/cases.jsonl with all five required fields. Run load_cases() and validate_cases() from scripts/run_evals.py to verify your addition passes schema validation before committing.

What happens if a case has an invalid risk level?

The validate_cases() function rejects any case where risk is not exactly low, medium, or high. The error is returned in the validation errors list, and the case will not be included in evaluation runs.

Can I modify the weights used in the release gate decision?

Yes. The WEIGHTS dictionary is defined in scripts/run_evals.py. Adjust these values to change how each metric contributes to the final weighted score, then re‑run summarize_scores() to see updated results.

Where are evaluation results stored during a run?

The --output flag for the run command specifies the destination path (typically results/<condition>.jsonl). These files contain raw score rows before aggregation by summarize_scores().

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}