# Data Validation in the i-have-adhd Project: Schema Enforcement and Cross-Condition Integrity

> Discover how the i-have-adhd project ensures data integrity through schema enforcement and cross-condition checks in its Python scripts. Learn about their robust validation process.

- Repository: [Ayoub Ghriss/i-have-adhd](https://github.com/ayghri/i-have-adhd)
- Tags: deep-dive
- Published: 2026-07-30

---

**The i-have-adhd project implements strict data validation in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) through three dedicated functions—`validate_cases`, `_validate_score`, and `_check_pairing`—that enforce JSON schema compliance, numeric range constraints, and cross-condition pairing integrity before aggregating evaluation results.**

The ayghri/i-have-adhd repository depends on rigorous data validation to maintain the integrity of its evaluation pipeline for ADHD-focused skills. By validating case catalogs and score rows at multiple entry points, the project ensures that only well-formed, consistent data reaches the aggregation stage, preventing downstream errors in comparative analysis.

## Core Validation Functions in scripts/run_evals.py

The validation logic centers on three specific functions defined in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py), each targeting a distinct layer of the evaluation data.

### Case Catalog Schema Validation (validate_cases)

The `validate_cases` function (lines 58‑79) enforces structural requirements on the JSON‑L case catalog, typically loaded from `evals/cases.jsonl`. This validator checks for:

- **Required fields**: Every case must contain `id`, `category`, `prompt`, `risk`, and `criteria`
- **Unique identifiers**: The `id` field must be a non‑empty string and unique across the catalog
- **Risk level constraints**: The `risk` field must be one of `low`, `medium`, or `high`
- **Criteria presence**: The `criteria` field must be a non‑empty list

If any case violates these rules, the function collects error messages and returns them for reporting.

### Score Row Integrity Checks (_validate_score)

After cases pass validation, the `_validate_score` function (lines 82‑95) validates individual score rows produced during evaluation runs. This function verifies:

- **Field completeness**: Presence of `case_id`, `trial`, `condition`, and all weighted metrics (`correctness`, `autonomy`, `actionability`, `safety`, `concision`), plus `blocker` and `notes`
- **Condition values**: The `condition` field must be one of `baseline`, `candidate`, or `comparator`
- **Metric ranges**: All weighted metric values must be numeric and fall within the 1‑5 range
- **Boolean typing**: The `blocker` field must be a boolean value

This function raises a `ValueError` immediately upon detecting a violation, halting the process before corrupted data enters the aggregation phase.

### Cross-Condition Pairing Validation (_check_pairing)

To ensure valid comparative analysis, the `_check_pairing` function (lines 101‑127) verifies that every condition (`baseline`, `candidate`, and optional `comparator`) contains judgments for the exact same set of `(case_id, trial)` combinations. The function raises errors for:

- Missing rows in any condition
- Duplicate entries within a condition
- Unmatched cases across conditions

This guarantees that statistical comparisons between baseline and candidate runs are based on identical inputs.

## Validation Entry Points and Workflow

The i-have-adhd project triggers data validation through two distinct pathways, ensuring coverage both during development and production runs.

### Command-Line Validation Mode

Developers can validate case catalogs without running full evaluations using the CLI command:

```bash
python scripts/run_evals.py validate

```

This command loads the case catalog via `load_cases` and executes `validate_cases`. Any validation errors print to stderr, and the process exits with a non‑zero status code, making it suitable for CI/CD pipelines.

### Runtime Validation During Evaluations

During active evaluation runs, the `run_evaluations` function first calls `validate_cases` on the loaded catalog. Subsequently, as the system generates score rows, each row passes through `_validate_score` before aggregation. This dual-layer approach ensures that both input data and generated results meet quality standards.

## Practical Validation Examples

The following examples demonstrate how to use the validation functions directly in Python scripts.

### Validating a Case Catalog Manually

```python

# Example: manually validating a list of case dictionaries

from scripts.run_evals import validate_cases

cases = [
    {
        "id": "example-1",
        "category": "demo",
        "prompt": "Do something",
        "risk": "low",
        "criteria": ["criterion A", "criterion B"],
    },
    # ... more cases ...

]

errors = validate_cases(cases)
if errors:
    for err in errors:
        print("Validation error:", err)
else:
    print("All cases are valid!")

```

### Validating Individual Score Rows

```python

# Example: validating a single score row before aggregation

from scripts.run_evals import _validate_score

score_row = {
    "case_id": "example-1",
    "trial": 1,
    "condition": "baseline",
    "correctness": 5,
    "autonomy": 4,
    "actionability": 5,
    "safety": 5,
    "concision": 4,
    "blocker": False,
    "notes": "Looks good",
}

# Will raise a ValueError if any rule is violated

_validate_score(score_row, index=1)
print("Score row is valid.")

```

## Testing the Validation Logic

The test suite in [`tests/test_run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/tests/test_run_evals.py) provides comprehensive coverage of the validation functions. These unit tests confirm that the system correctly catches:

- Missing required fields in case definitions
- Duplicate `id` values within the case catalog
- Invalid `risk` levels outside the allowed enumeration
- Malformed score rows with non‑numeric metrics or out‑of‑range values

## Summary

- **Schema enforcement**: The `validate_cases` function in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) (lines 58‑79) ensures case catalogs contain required fields with valid risk levels and unique IDs.
- **Row-level validation**: The `_validate_score` function (lines 82‑95) guarantees score metrics are numeric, within 1‑5 ranges, and contain valid condition identifiers.
- **Pairing integrity**: The `_check_pairing` function (lines 101‑127) verifies identical `(case_id, trial)` coverage across baseline, candidate, and comparator conditions.
- **Dual entry points**: Validation runs via `run_evals.py validate` for standalone checks and automatically during `run_evaluations` execution.
- **Test coverage**: Unit tests in [`tests/test_run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/tests/test_run_evals.py) prevent regression in validation logic.

## Frequently Asked Questions

### What specific fields are required in the case catalog for validation to pass?

The `validate_cases` function requires every case to include `id` (unique non‑empty string), `category`, `prompt`, `risk` (must be `low`, `medium`, or `high`), and `criteria` (non‑empty list). Missing any of these fields or violating the value constraints will generate validation errors.

### How does the project ensure score row metrics stay within valid ranges?

The `_validate_score` function explicitly checks that weighted metrics (`correctness`, `autonomy`, `actionability`, `safety`, `concision`) are numeric values between 1 and 5. If a metric falls outside this range or contains non‑numeric data, the function raises a `ValueError` and halts processing.

### What happens if the evaluation data contains mismatched conditions across trials?

The `_check_pairing` function (lines 101‑127) detects missing, duplicate, or unmatched `(case_id, trial)` rows across conditions. When it finds discrepancies between baseline, candidate, or comparator sets, it raises an error preventing aggregation on incomplete or inconsistent data.

### Can I validate my case catalog without running the full evaluation suite?

Yes. The repository provides a dedicated command-line validation mode. Running `python scripts/run_evals.py validate` loads the default catalog from `evals/cases.jsonl` and executes all schema checks without triggering actual model evaluations, returning a non‑zero exit code if validation fails.