# How to Run Evals Locally Using run_evals.py Validate

> Learn how to run evals locally with run_evals.py validate. Verify your JSON-Lines catalog schema without external connections. Perfect for local testing and validation.

- Repository: [Ayoub Ghriss/i-have-adhd](https://github.com/ayghri/i-have-adhd)
- Tags: how-to-guide
- Published: 2026-08-22

---

**To run evals locally using `run_evals.py validate`, execute the `validate` sub-command from the repository root to verify your JSON-Lines case catalog meets schema requirements without connecting to external services.**

The `ayghri/i-have-adhd` repository includes a lightweight CLI driver at [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) that manages AI safety evaluation workflows. The `validate` sub-command provides offline schema verification for your evaluation cases, ensuring required fields, unique identifiers, and proper risk levels are present before you execute any model runs.

## How run_evals.py Validate Works

The validation script is a self-contained Python module with a shebang line, allowing direct execution or invocation via the Python interpreter. Located at [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py), the tool loads and parses evaluation case catalogs using internal helper functions.

When you invoke the `validate` sub-command, the script executes `read_jsonl` (lines 29-41) to parse the JSON-Lines entries, then passes the data to `validate_cases` (lines 59-79). This function performs schema checks including:

- **Required field presence** – Ensures all mandatory keys exist in each case entry
- **Unique ID validation** – Confirms no duplicate identifiers across the catalog
- **Risk level verification** – Validates that risk classifications conform to expected schema values
- **Non-empty criteria** – Checks that evaluation criteria fields contain substantive content

If validation fails, the script writes error messages to `stderr` and exits with a non-zero status code. Successful validation prints "Evaluation cases are valid." (lines 42-50). This process requires no API keys or network connectivity, making it safe for air-gapped environments.

## Prerequisites and Setup

Clone the repository and navigate to the project root:

```bash
git clone https://github.com/ayghri/i-have-adhd.git
cd i-have-adhd

```

Optionally make the script executable:

```bash
chmod +x scripts/run_evals.py

```

The default validation target is `evals/cases.jsonl` (defined at lines 11-13), though you can specify custom paths.

## Running Basic Validation

To validate the default case catalog, run:

```bash
./scripts/run_evals.py validate

```

Or equivalently using the Python interpreter:

```bash
python3 scripts/run_evals.py validate

```

This command loads `evals/cases.jsonl` and runs the full validation suite locally.

## Validating Custom Case Files

To validate a custom evaluation catalog, provide the `--cases` argument:

```bash
python3 scripts/run_evals.py validate --cases path/to/my_cases.jsonl

```

The script accepts any JSON-Lines file following the repository's case schema, making it flexible for testing experimental evaluation sets before integration.

## Understanding Output Messages

The `validate` sub-command provides clear exit status signaling:

- **Success**: Prints `Evaluation cases are valid.` and exits with status code `0`
- **Failure**: Writes `ERROR:` messages to `stderr` describing specific schema violations (missing fields, duplicate IDs, malformed risk levels) and exits with non-zero status

This exit behavior allows seamless integration into CI/CD pipelines, where non-zero exits will halt workflows if evaluation catalogs are malformed.

## Summary

- **Offline Operation**: The `validate` sub-command requires no external services, functioning entirely within the local Python environment.
- **Schema Enforcement**: Uses `validate_cases` to verify required fields, unique IDs, risk levels, and non-empty criteria in `evals/cases.jsonl` or custom files.
- **Flexible Invocation**: Execute directly via shebang ([`./scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/./scripts/run_evals.py)) or interpreter (`python3 scripts/run_evals.py`).
- **CI Integration**: Exit codes and `stderr` reporting enable automated validation gates in deployment pipelines.

## Frequently Asked Questions

### Do I need API keys to run run_evals.py validate?

No. The `validate` sub-command performs local schema checking only using the `validate_cases` function. It does not instantiate model clients or make network requests, making it suitable for offline development and pre-deployment checks.

### What specific checks does the validator perform on each case?

According to the source code in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) (lines 59-79), the `validate_cases` function checks for required fields, ensures ID uniqueness across the catalog, validates that risk levels conform to expected values, and verifies that evaluation criteria fields are not empty.

### Can I validate multiple case files at once?

The current implementation accepts a single `--cases` argument pointing to one JSON-Lines file. To validate multiple files, invoke the script separately for each path or concatenate your case catalogs into a single JSON-Lines file before validation.

### What exit code does run_evals.py validate return on failure?

The script exits with a non-zero status code when validation fails, writing specific error descriptions to `stderr`. On success, it exits with code `0` and prints "Evaluation cases are valid." This behavior supports shell scripting and automated workflow integration.