# What Is the Format of Test Cases for i-have-adhd? Complete Schema Guide

> Learn the i-have-adhd test case format. Discover the required JSON Lines schema including id, category, prompt, risk, and criteria for effective testing.

- Repository: [Ayoub Ghriss/i-have-adhd](https://github.com/ayghri/i-have-adhd)
- Tags: api-reference
- Published: 2026-08-31

---

**The i-have-adhd repository uses JSON Lines format for test cases, with each line containing a JSON object that requires five mandatory fields: `id`, `category`, `prompt`, `risk`, and `criteria`.**

This open-source evaluation framework for ADHD-focused AI interactions stores its test cases in `evals/cases.jsonl`. Understanding the exact schema is essential for contributors adding new evaluation scenarios or integrating the framework into custom pipelines. Below is the complete breakdown of the test case format, validation logic, and working code examples drawn directly from the source.

## JSON Lines Structure and Required Fields

The test case file uses **JSON Lines** (`.jsonl`) format—one JSON object per line, no trailing commas, no outer array wrapper. This design enables streaming reads and append-only updates without rewriting the entire file.

Each test case must include these five fields:

| Field | Type | Constraints | Purpose |
|-------|------|-------------|---------|
| `id` | string | Non-empty, unique across all cases | Permanent identifier for tracking and deduplication |
| `category` | string | No enforced enum, but common values include `direct-answer`, `coding`, `safety` | Logical grouping for analysis and filtering |
| `prompt` | string | Non-empty | The actual user query or task presented to the model |
| `risk` | string enum | Must be `"low"`, `"medium"`, or `"high"` | Triggers appropriate safety handling mechanisms |
| `criteria` | array of strings | Non-empty array, each element non-empty | Rubric for automated or manual evaluation of responses |

### Example Test Case

```json
{
  "id": "code-answer",
  "category": "coding",
  "prompt": "Write a TypeScript function isEven(n: number): boolean. Return only the code block.",
  "risk": "low",
  "criteria": [
    "Returns correct TypeScript.",
    "Follows the requested output-only format."
  ]
}

```

Notice the flat structure—no nested objects, no metadata fields beyond the core five. This minimalism keeps validation fast and parsing predictable.

## Validation Logic in run_evals.py

The [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) file enforces schema compliance through the `validate_cases` function (lines 59-79). This function performs four critical checks before any evaluation proceeds:

- **Missing field detection** — ensures all five required keys exist
- **Duplicate ID detection** — builds a set of seen IDs and raises errors on collisions
- **Risk value validation** — confirms `risk` is one of the three allowed strings
- **Criteria non-emptiness** — verifies the `criteria` array exists and contains at least one non-empty string

```python

# From scripts/run_evals.py (simplified representation)

def validate_cases(cases: list[dict]) -> list[str]:
    """Return list of validation error messages, empty if valid."""
    errors = []
    seen_ids = set()
    
    for case in cases:
        # Check required fields

        for field in ["id", "category", "prompt", "risk", "criteria"]:
            if field not in case:
                errors.append(f"Case missing '{field}'")
        
        # Validate risk enum

        if case.get("risk") not in ("low", "medium", "high"):
            errors.append(f"Invalid risk value: {case.get('risk')}")
        
        # Check for duplicate IDs

        case_id = case.get("id")
        if case_id in seen_ids:
            errors.append(f"Duplicate id: {case_id}")
        seen_ids.add(case_id)
        
        # Validate criteria

        criteria = case.get("criteria", [])
        if not criteria or not all(criteria):
            errors.append(f"Empty criteria for case: {case_id}")
    
    return errors

```

Loading cases follows the same pattern used in production evaluation runs:

```python
from pathlib import Path
from scripts.run_evals import load_cases, validate_cases

cases_path = Path(__file__).parent / "evals" / "cases.jsonl"
cases = load_cases(cases_path)

errors = validate_cases(cases)
if errors:
    raise RuntimeError(f"Invalid test cases: {errors}")

# Each case is a dict with the required keys

for case in cases:
    print(f"{case['id']} ({case['category']}): {case['prompt']}")

```

The `load_cases` function handles JSON parsing and basic line-level error recovery, while `validate_cases` performs semantic validation.

## Key Source Files and Their Roles

| File | Purpose | Line References |
|------|---------|---------------|
| `evals/cases.jsonl` | Actual test case data | All test cases stored here |
| [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) | Core validation and scoring logic | `validate_cases` at lines 59-79 |
| [`tests/test_run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/tests/test_run_evals.py) | Unit tests for loading, validation, scoring | Covers edge cases in schema enforcement |
| [`tests/test_opencode_plugin.py`](https://github.com/ayghri/i-have-adhd/blob/main/tests/test_opencode_plugin.py) | Integration tests using real case format | Validates end-to-end evaluation flow |

The test suite in [`tests/test_run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/tests/test_run_evals.py) deliberately includes malformed cases to verify that `validate_cases` catches each failure mode—missing fields, invalid risk levels, and empty criteria arrays.

## Risk Levels and Their Semantic Meaning

The `risk` field isn't merely categorical; it drives downstream behavior in the evaluation pipeline:

- **`"low"`** — Standard task execution with minimal safety constraints
- **`"medium"`** — Enhanced review triggers, potentially human-in-the-loop sampling
- **`"high"`** — Maximum scrutiny, may block auto-approval regardless of criteria satisfaction

This three-tier system aligns with the repository's focus on responsible AI deployment for ADHD-related applications, where certain prompt categories (medical advice, medication discussions) warrant elevated caution.

## Criteria Design Best Practices

The `criteria` array functions as a **scoring rubric**. Each string should represent an independently verifiable condition. Well-designed criteria are:

- **Atomic** — one concept per string
- **Observable** — evaluators can confirm satisfaction without inference
- **Complete** — collectively cover all success dimensions

Avoid compound criteria like "Code is correct and well-documented"—split into separate items for testability.

## Summary

- The **i-have-adhd test case format** is JSON Lines with five mandatory fields: `id`, `category`, `prompt`, `risk`, and `criteria`
- Validation occurs in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) through the `validate_cases` function
- Risk levels (`low`/`medium`/`high`) influence safety handling beyond pure evaluation scoring
- Criteria arrays must be non-empty and contain verifiable, atomic conditions
- All validation errors surface before any model evaluation begins, preventing runtime surprises

## Frequently Asked Questions

### What happens if a test case is missing the `risk` field?

The `validate_cases` function in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) adds an error message to its return list stating the field is missing. Evaluation will not proceed until all validation errors are resolved. The function treats missing required fields as blocking errors.

### Can I add custom fields beyond the five required ones?

While the JSON parser will accept additional keys, the validation logic ignores them. Future versions of the framework may reject unknown fields, so contributors should stick to the documented schema. Store auxiliary metadata externally if needed.

### How does the framework handle duplicate IDs across the file?

`validate_cases` builds a set of encountered IDs and flags any duplicates as validation errors. Each `id` must be globally unique within `evals/cases.jsonl`. The error message includes the conflicting ID for manual resolution.

### Is there a maximum recommended length for the `prompt` field?

The schema imposes no hard limit, but practical constraints apply. Very long prompts may hit model context windows or complicate diff-based evaluation. The reference cases typically stay under 500 tokens. Consider splitting complex multi-part scenarios into separate test cases with linked IDs.