Where to Find the Evaluation Data for i-have-adhd: Complete File Guide

All evaluation data for the i-have-adhd project resides in the top-level evals/ directory, which contains the case catalog, scoring rubric, runner configuration, and orchestration scripts.

The ayghri/i-have-adhd repository maintains a comprehensive evaluation framework for assessing language model behaviors. If you need to locate the evaluation data for i-have-adhd, you will find all assets organized under the root-level evals/ directory. This folder houses the complete test suite including case definitions, judgment criteria, and execution configurations.

Evaluation Data Location and Structure

The evaluation data for i-have-adhd is centralized in the evals/ directory at the repository root. This location serves as the single source of truth for all testing assets, containing structured test scenarios, evaluation criteria, and supporting documentation. The directory works in conjunction with scripts/run_evals.py, which provides the command-line interface for validation, planning, execution, and scoring.

Core Evaluation Files

The evals/ directory contains four primary files that constitute the complete evaluation dataset.

Case Catalog (evals/cases.jsonl)

The evals/cases.jsonl file stores the evaluation cases in JSON-Lines format. Each line represents a distinct test scenario containing the prompt, risk assessment, evaluation criteria, and unique case identifiers. This file acts as the master catalog that drives the evaluation harness.

Scoring Rubric (evals/rubric.md)

The evals/rubric.md file defines the scoring contract used by human judges to evaluate model responses. This document specifies correctness criteria, safety guidelines, and judgment standards for assessing outputs against the test cases.

Runner Configuration (evals/runners.example.json)

The evals/runners.example.json file provides a template for configuring language model runners such as Claude. It includes command specifications, response format requirements, and budget flags necessary to integrate external models into the evaluation pipeline.

Documentation (evals/README.md)

The evals/README.md file contains high-level documentation describing the evaluation workflow, directory structure, and usage instructions for researchers and contributors.

Running Evaluations with scripts/run_evals.py

The scripts/run_evals.py script orchestrates the evaluation pipeline through four primary commands: validate, plan, run, and score. This Python CLI tool processes the evaluation data from the evals/ directory and manages the execution lifecycle.

To validate the case catalog and ensure all entries contain required fields with no duplicate IDs:

python3 scripts/run_evals.py validate

To generate a run matrix showing the (case_id, trial, condition) combinations that will be executed:

python3 scripts/run_evals.py plan --trials 3 --include-comparator

To execute evaluations against a specific model runner:

python3 scripts/run_evals.py run \
    --runner-config evals/runners.example.json \
    --runner claude \
    --condition baseline \
    --condition-skill skills/i-have-adhd/SKILL.md \
    --trials 3 \
    --budget-usd 12.50 \
    --output evals/results/responses.jsonl

To aggregate manually-judged scores and generate a summary report:

python3 scripts/run_evals.py score evals/results/scores.jsonl

Summary

• The evaluation data for i-have-adhd is located in the evals/ directory at the repository root.
• evals/cases.jsonl contains the JSON-Lines catalog of test scenarios with prompts and criteria.
• evals/rubric.md defines the scoring standards for human judges.
• evals/runners.example.json provides configuration templates for model runners like Claude.
• scripts/run_evals.py serves as the CLI harness for validating, planning, running, and scoring evaluations.

Frequently Asked Questions

Where is the evaluation data stored in the i-have-adhd repository?

All evaluation data resides in the top-level evals/ directory within the ayghri/i-have-adhd repository. This folder contains the case catalog, scoring rubric, runner configuration examples, and workflow documentation.

What format are the evaluation cases in?

The evaluation cases are stored in evals/cases.jsonl using JSON-Lines format, where each line represents a separate test case containing the prompt, risk assessment, and evaluation criteria.

How do I run the evaluation harness?

Execute scripts/run_evals.py with subcommands validate, plan, run, or score. For example, use python3 scripts/run_evals.py validate to check the case catalog integrity, or python3 scripts/run_evals.py run with appropriate flags to execute tests against configured model runners.

What is the purpose of the rubric.md file?

The evals/rubric.md file establishes the scoring contract that human judges use to evaluate model responses on correctness, safety, and other criteria defined within the evaluation framework.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}