What Is the Format of Test Cases for i-have-adhd? Complete Schema Guide

The i-have-adhd repository uses JSON Lines format for test cases, with each line containing a JSON object that requires five mandatory fields: id, category, prompt, risk, and criteria.

This open-source evaluation framework for ADHD-focused AI interactions stores its test cases in evals/cases.jsonl. Understanding the exact schema is essential for contributors adding new evaluation scenarios or integrating the framework into custom pipelines. Below is the complete breakdown of the test case format, validation logic, and working code examples drawn directly from the source.

JSON Lines Structure and Required Fields

The test case file uses JSON Lines (.jsonl) format—one JSON object per line, no trailing commas, no outer array wrapper. This design enables streaming reads and append-only updates without rewriting the entire file.

Each test case must include these five fields:

Field	Type	Constraints	Purpose
id	string	Non-empty, unique across all cases	Permanent identifier for tracking and deduplication
category	string	No enforced enum, but common values include direct-answer, coding, safety	Logical grouping for analysis and filtering
prompt	string	Non-empty	The actual user query or task presented to the model
risk	string enum	Must be "low", "medium", or "high"	Triggers appropriate safety handling mechanisms
criteria	array of strings	Non-empty array, each element non-empty	Rubric for automated or manual evaluation of responses

Example Test Case

{
  "id": "code-answer",
  "category": "coding",
  "prompt": "Write a TypeScript function isEven(n: number): boolean. Return only the code block.",
  "risk": "low",
  "criteria": [
    "Returns correct TypeScript.",
    "Follows the requested output-only format."
  ]
}

Notice the flat structure—no nested objects, no metadata fields beyond the core five. This minimalism keeps validation fast and parsing predictable.

Validation Logic in run_evals.py

The scripts/run_evals.py file enforces schema compliance through the validate_cases function (lines 59-79). This function performs four critical checks before any evaluation proceeds:

• Missing field detection — ensures all five required keys exist
• Duplicate ID detection — builds a set of seen IDs and raises errors on collisions
• Risk value validation — confirms risk is one of the three allowed strings
• Criteria non-emptiness — verifies the criteria array exists and contains at least one non-empty string

# From scripts/run_evals.py (simplified representation)

def validate_cases(cases: list[dict]) -> list[str]:
    """Return list of validation error messages, empty if valid."""
    errors = []
    seen_ids = set()
    
    for case in cases:
        # Check required fields

        for field in ["id", "category", "prompt", "risk", "criteria"]:
            if field not in case:
                errors.append(f"Case missing '{field}'")
        
        # Validate risk enum

        if case.get("risk") not in ("low", "medium", "high"):
            errors.append(f"Invalid risk value: {case.get('risk')}")
        
        # Check for duplicate IDs

        case_id = case.get("id")
        if case_id in seen_ids:
            errors.append(f"Duplicate id: {case_id}")
        seen_ids.add(case_id)
        
        # Validate criteria

        criteria = case.get("criteria", [])
        if not criteria or not all(criteria):
            errors.append(f"Empty criteria for case: {case_id}")
    
    return errors

Loading cases follows the same pattern used in production evaluation runs:

from pathlib import Path
from scripts.run_evals import load_cases, validate_cases

cases_path = Path(__file__).parent / "evals" / "cases.jsonl"
cases = load_cases(cases_path)

errors = validate_cases(cases)
if errors:
    raise RuntimeError(f"Invalid test cases: {errors}")

# Each case is a dict with the required keys

for case in cases:
    print(f"{case['id']} ({case['category']}): {case['prompt']}")

The load_cases function handles JSON parsing and basic line-level error recovery, while validate_cases performs semantic validation.

Key Source Files and Their Roles

File	Purpose	Line References
evals/cases.jsonl	Actual test case data	All test cases stored here
scripts/run_evals.py	Core validation and scoring logic	validate_cases at lines 59-79
tests/test_run_evals.py	Unit tests for loading, validation, scoring	Covers edge cases in schema enforcement
tests/test_opencode_plugin.py	Integration tests using real case format	Validates end-to-end evaluation flow

The test suite in tests/test_run_evals.py deliberately includes malformed cases to verify that validate_cases catches each failure mode—missing fields, invalid risk levels, and empty criteria arrays.

Risk Levels and Their Semantic Meaning

The risk field isn't merely categorical; it drives downstream behavior in the evaluation pipeline:

• "low" — Standard task execution with minimal safety constraints
• "medium" — Enhanced review triggers, potentially human-in-the-loop sampling
• "high" — Maximum scrutiny, may block auto-approval regardless of criteria satisfaction

This three-tier system aligns with the repository's focus on responsible AI deployment for ADHD-related applications, where certain prompt categories (medical advice, medication discussions) warrant elevated caution.

Criteria Design Best Practices

The criteria array functions as a scoring rubric. Each string should represent an independently verifiable condition. Well-designed criteria are:

• Atomic — one concept per string
• Observable — evaluators can confirm satisfaction without inference
• Complete — collectively cover all success dimensions

Avoid compound criteria like "Code is correct and well-documented"—split into separate items for testability.

Summary

• The i-have-adhd test case format is JSON Lines with five mandatory fields: id, category, prompt, risk, and criteria
• Validation occurs in scripts/run_evals.py through the validate_cases function
• Risk levels (low/medium/high) influence safety handling beyond pure evaluation scoring
• Criteria arrays must be non-empty and contain verifiable, atomic conditions
• All validation errors surface before any model evaluation begins, preventing runtime surprises

Frequently Asked Questions

What happens if a test case is missing the risk field?

The validate_cases function in scripts/run_evals.py adds an error message to its return list stating the field is missing. Evaluation will not proceed until all validation errors are resolved. The function treats missing required fields as blocking errors.

Can I add custom fields beyond the five required ones?

While the JSON parser will accept additional keys, the validation logic ignores them. Future versions of the framework may reject unknown fields, so contributors should stick to the documented schema. Store auxiliary metadata externally if needed.

How does the framework handle duplicate IDs across the file?

validate_cases builds a set of encountered IDs and flags any duplicates as validation errors. Each id must be globally unique within evals/cases.jsonl. The error message includes the conflicting ID for manual resolution.

Is there a maximum recommended length for the prompt field?

The schema imposes no hard limit, but practical constraints apply. Very long prompts may hit model context windows or complicate diff-based evaluation. The reference cases typically stay under 500 tokens. Consider splitting complex multi-part scenarios into separate test cases with linked IDs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}