Data Validation in the i-have-adhd Project: Schema Enforcement and Cross-Condition Integrity

The i-have-adhd project implements strict data validation in scripts/run_evals.py through three dedicated functions—validate_cases, _validate_score, and _check_pairing—that enforce JSON schema compliance, numeric range constraints, and cross-condition pairing integrity before aggregating evaluation results.

The ayghri/i-have-adhd repository depends on rigorous data validation to maintain the integrity of its evaluation pipeline for ADHD-focused skills. By validating case catalogs and score rows at multiple entry points, the project ensures that only well-formed, consistent data reaches the aggregation stage, preventing downstream errors in comparative analysis.

Core Validation Functions in scripts/run_evals.py

The validation logic centers on three specific functions defined in scripts/run_evals.py, each targeting a distinct layer of the evaluation data.

Case Catalog Schema Validation (validate_cases)

The validate_cases function (lines 58‑79) enforces structural requirements on the JSON‑L case catalog, typically loaded from evals/cases.jsonl. This validator checks for:

• Required fields: Every case must contain id, category, prompt, risk, and criteria
• Unique identifiers: The id field must be a non‑empty string and unique across the catalog
• Risk level constraints: The risk field must be one of low, medium, or high
• Criteria presence: The criteria field must be a non‑empty list

If any case violates these rules, the function collects error messages and returns them for reporting.

Score Row Integrity Checks (_validate_score)

After cases pass validation, the _validate_score function (lines 82‑95) validates individual score rows produced during evaluation runs. This function verifies:

• Field completeness: Presence of case_id, trial, condition, and all weighted metrics (correctness, autonomy, actionability, safety, concision), plus blocker and notes
• Condition values: The condition field must be one of baseline, candidate, or comparator
• Metric ranges: All weighted metric values must be numeric and fall within the 1‑5 range
• Boolean typing: The blocker field must be a boolean value

This function raises a ValueError immediately upon detecting a violation, halting the process before corrupted data enters the aggregation phase.

Cross-Condition Pairing Validation (_check_pairing)

To ensure valid comparative analysis, the _check_pairing function (lines 101‑127) verifies that every condition (baseline, candidate, and optional comparator) contains judgments for the exact same set of (case_id, trial) combinations. The function raises errors for:

• Missing rows in any condition
• Duplicate entries within a condition
• Unmatched cases across conditions

This guarantees that statistical comparisons between baseline and candidate runs are based on identical inputs.

Validation Entry Points and Workflow

The i-have-adhd project triggers data validation through two distinct pathways, ensuring coverage both during development and production runs.

Command-Line Validation Mode

Developers can validate case catalogs without running full evaluations using the CLI command:

python scripts/run_evals.py validate

This command loads the case catalog via load_cases and executes validate_cases. Any validation errors print to stderr, and the process exits with a non‑zero status code, making it suitable for CI/CD pipelines.

Runtime Validation During Evaluations

During active evaluation runs, the run_evaluations function first calls validate_cases on the loaded catalog. Subsequently, as the system generates score rows, each row passes through _validate_score before aggregation. This dual-layer approach ensures that both input data and generated results meet quality standards.

Practical Validation Examples

The following examples demonstrate how to use the validation functions directly in Python scripts.

Validating a Case Catalog Manually

# Example: manually validating a list of case dictionaries

from scripts.run_evals import validate_cases

cases = [
    {
        "id": "example-1",
        "category": "demo",
        "prompt": "Do something",
        "risk": "low",
        "criteria": ["criterion A", "criterion B"],
    },
    # ... more cases ...

]

errors = validate_cases(cases)
if errors:
    for err in errors:
        print("Validation error:", err)
else:
    print("All cases are valid!")

Validating Individual Score Rows

# Example: validating a single score row before aggregation

from scripts.run_evals import _validate_score

score_row = {
    "case_id": "example-1",
    "trial": 1,
    "condition": "baseline",
    "correctness": 5,
    "autonomy": 4,
    "actionability": 5,
    "safety": 5,
    "concision": 4,
    "blocker": False,
    "notes": "Looks good",
}

# Will raise a ValueError if any rule is violated

_validate_score(score_row, index=1)
print("Score row is valid.")

Testing the Validation Logic

The test suite in tests/test_run_evals.py provides comprehensive coverage of the validation functions. These unit tests confirm that the system correctly catches:

• Missing required fields in case definitions
• Duplicate id values within the case catalog
• Invalid risk levels outside the allowed enumeration
• Malformed score rows with non‑numeric metrics or out‑of‑range values

Summary

• Schema enforcement: The validate_cases function in scripts/run_evals.py (lines 58‑79) ensures case catalogs contain required fields with valid risk levels and unique IDs.
• Row-level validation: The _validate_score function (lines 82‑95) guarantees score metrics are numeric, within 1‑5 ranges, and contain valid condition identifiers.
• Pairing integrity: The _check_pairing function (lines 101‑127) verifies identical (case_id, trial) coverage across baseline, candidate, and comparator conditions.
• Dual entry points: Validation runs via run_evals.py validate for standalone checks and automatically during run_evaluations execution.
• Test coverage: Unit tests in tests/test_run_evals.py prevent regression in validation logic.

Frequently Asked Questions

What specific fields are required in the case catalog for validation to pass?

The validate_cases function requires every case to include id (unique non‑empty string), category, prompt, risk (must be low, medium, or high), and criteria (non‑empty list). Missing any of these fields or violating the value constraints will generate validation errors.

How does the project ensure score row metrics stay within valid ranges?

The _validate_score function explicitly checks that weighted metrics (correctness, autonomy, actionability, safety, concision) are numeric values between 1 and 5. If a metric falls outside this range or contains non‑numeric data, the function raises a ValueError and halts processing.

What happens if the evaluation data contains mismatched conditions across trials?

The _check_pairing function (lines 101‑127) detects missing, duplicate, or unmatched (case_id, trial) rows across conditions. When it finds discrepancies between baseline, candidate, or comparator sets, it raises an error preventing aggregation on incomplete or inconsistent data.

Can I validate my case catalog without running the full evaluation suite?

Yes. The repository provides a dedicated command-line validation mode. Running python scripts/run_evals.py validate loads the default catalog from evals/cases.jsonl and executes all schema checks without triggering actual model evaluations, returning a non‑zero exit code if validation fails.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}