How to Run Evals Locally Using run_evals.py Validate

To run evals locally using run_evals.py validate, execute the validate sub-command from the repository root to verify your JSON-Lines case catalog meets schema requirements without connecting to external services.

The ayghri/i-have-adhd repository includes a lightweight CLI driver at scripts/run_evals.py that manages AI safety evaluation workflows. The validate sub-command provides offline schema verification for your evaluation cases, ensuring required fields, unique identifiers, and proper risk levels are present before you execute any model runs.

How run_evals.py Validate Works

The validation script is a self-contained Python module with a shebang line, allowing direct execution or invocation via the Python interpreter. Located at scripts/run_evals.py, the tool loads and parses evaluation case catalogs using internal helper functions.

When you invoke the validate sub-command, the script executes read_jsonl (lines 29-41) to parse the JSON-Lines entries, then passes the data to validate_cases (lines 59-79). This function performs schema checks including:

• Required field presence – Ensures all mandatory keys exist in each case entry
• Unique ID validation – Confirms no duplicate identifiers across the catalog
• Risk level verification – Validates that risk classifications conform to expected schema values
• Non-empty criteria – Checks that evaluation criteria fields contain substantive content

If validation fails, the script writes error messages to stderr and exits with a non-zero status code. Successful validation prints "Evaluation cases are valid." (lines 42-50). This process requires no API keys or network connectivity, making it safe for air-gapped environments.

Prerequisites and Setup

Clone the repository and navigate to the project root:

git clone https://github.com/ayghri/i-have-adhd.git
cd i-have-adhd

Optionally make the script executable:

chmod +x scripts/run_evals.py

The default validation target is evals/cases.jsonl (defined at lines 11-13), though you can specify custom paths.

Running Basic Validation

To validate the default case catalog, run:

./scripts/run_evals.py validate

Or equivalently using the Python interpreter:

python3 scripts/run_evals.py validate

This command loads evals/cases.jsonl and runs the full validation suite locally.

Validating Custom Case Files

To validate a custom evaluation catalog, provide the --cases argument:

python3 scripts/run_evals.py validate --cases path/to/my_cases.jsonl

The script accepts any JSON-Lines file following the repository's case schema, making it flexible for testing experimental evaluation sets before integration.

Understanding Output Messages

The validate sub-command provides clear exit status signaling:

• Success: Prints Evaluation cases are valid. and exits with status code 0
• Failure: Writes ERROR: messages to stderr describing specific schema violations (missing fields, duplicate IDs, malformed risk levels) and exits with non-zero status

This exit behavior allows seamless integration into CI/CD pipelines, where non-zero exits will halt workflows if evaluation catalogs are malformed.

Summary

• Offline Operation: The validate sub-command requires no external services, functioning entirely within the local Python environment.
• Schema Enforcement: Uses validate_cases to verify required fields, unique IDs, risk levels, and non-empty criteria in evals/cases.jsonl or custom files.
• Flexible Invocation: Execute directly via shebang (./scripts/run_evals.py) or interpreter (python3 scripts/run_evals.py).
• CI Integration: Exit codes and stderr reporting enable automated validation gates in deployment pipelines.

Frequently Asked Questions

Do I need API keys to run run_evals.py validate?

No. The validate sub-command performs local schema checking only using the validate_cases function. It does not instantiate model clients or make network requests, making it suitable for offline development and pre-deployment checks.

What specific checks does the validator perform on each case?

According to the source code in scripts/run_evals.py (lines 59-79), the validate_cases function checks for required fields, ensures ID uniqueness across the catalog, validates that risk levels conform to expected values, and verifies that evaluation criteria fields are not empty.

Can I validate multiple case files at once?

The current implementation accepts a single --cases argument pointing to one JSON-Lines file. To validate multiple files, invoke the script separately for each path or concatenate your case catalogs into a single JSON-Lines file before validation.

What exit code does run_evals.py validate return on failure?

The script exits with a non-zero status code when validation fails, writing specific error descriptions to stderr. On success, it exits with code 0 and prints "Evaluation cases are valid." This behavior supports shell scripting and automated workflow integration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}