# Dependencies Required to Run the Evaluation Suite for 'i-have-adhd'

> Discover the minimal dependencies to run the i-have-adhd evaluation suite. Get started with Python 3.10+ and optional CLI runners, no extra Python packages needed.

- Repository: [Ayoub Ghriss/i-have-adhd](https://github.com/ayghri/i-have-adhd)
- Tags: api-reference
- Published: 2026-08-03

---

**The evaluation suite for `i-have-adhd` requires only Python 3.10+ and optional external CLI runners (Claude or Codex), with no third-party Python packages needed.**

The `i-have-adhd` repository includes a lightweight, pure-Python evaluation framework for testing LLM-assisted coding workflows. According to the source code at `ayghri/i-have-adhd`, the entire suite runs on the standard library and external CLI tools configured via JSON.

## Core Python Requirements

The evaluation suite is intentionally minimal. Everything needed to validate, plan, run, and score evaluations is included in Python's standard library.

**Python 3.10 or newer** is required because [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) uses modern type-hint syntax (`list[dict[str, Any]]`) and `pathlib.Path` objects that depend on recent interpreter features.

The script imports only standard library modules:
- `argparse` — CLI argument parsing
- `json` / `jsonlines` handling via built-in `json`
- `subprocess` — invoking external runners
- `pathlib` — filesystem operations

No [`requirements.txt`](https://github.com/ayghri/i-have-adhd/blob/main/requirements.txt) or [`pyproject.toml`](https://github.com/ayghri/i-have-adhd/blob/main/pyproject.toml) dependencies exist for the evaluation suite itself.

## Optional External Runners

While the Python code runs standalone, executing actual LLM evaluations requires external CLI tools defined in [`evals/runners.example.json`](https://github.com/ayghri/i-have-adhd/blob/main/evals/runners.example.json):

| Runner | Tool | Purpose |
|--------|------|---------|
| `claude` | [Claude CLI](https://claude.ai) | Evaluate using Anthropic's Claude model |
| `codex` | [Codex CLI](https://openai.com) | Evaluate using OpenAI's Codex model |

These are **optional**—the framework supports any runner configured in your local [`evals/runners.json`](https://github.com/ayghri/i-have-adhd/blob/main/evals/runners.json). You only need the one matching your target model.

## Installation and Verification

Clone the repository and verify your environment:

```bash
git clone https://github.com/ayghri/i-have-adhd.git
cd i-have-adhd

# Verify Python version

python3 --version  # Must be 3.10+

# Validate the evaluation suite works

python3 scripts/run_evals.py --help

```

### External Runner Installation (Optional)

Install Claude CLI:

```bash

# Via npm (requires Node.js)

npm install -g @anthropic-ai/claude-cli

```

Or install Codex CLI:

```bash

# Via pip

pip install openai-codex

```

Then copy and customize the runner configuration:

```bash
cp evals/runners.example.json evals/runners.json

# Edit evals/runners.json to match your installed tools

```

## Running the Evaluation Suite

### Validate Case Catalog

Check that your JSONL case files are well-formed before running evaluations:

```bash
python3 scripts/run_evals.py validate

```

This executes the `validate_cases()` function in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py).

### Plan Evaluation Matrix

Preview what will run without executing:

```bash
python3 scripts/run_evals.py plan \
  --trials 3 \
  --include-comparator

```

Outputs JSONL describing every case/trial/condition combination.

### Execute Evaluation

Run with Claude as the runner:

```bash
python3 scripts/run_evals.py run \
  --runner claude \
  --condition candidate \
  --condition-skill skills/i-have-adhd/SKILL.md \
  --output evals/results/responses.jsonl

```

Parameters match the argparse configuration in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py):
- `--runner` — key from [`evals/runners.json`](https://github.com/ayghri/i-have-adhd/blob/main/evals/runners.json)
- `--condition` — experimental condition to test
- `--condition-skill` — path to skill markdown file
- `--output` — JSONL destination for responses

### Score Results

Aggregate manually judged results:

```bash
python3 scripts/run_evals.py score evals/results/scores.jsonl

```

Invokes `summarize_scores()` to print performance metrics.

## Key Source Files

Understanding the dependencies requires examining these locations:

- **[`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py)** — Core CLI with `validate()`, `plan()`, `run()`, and `score()` subcommands
- **[`evals/runners.example.json`](https://github.com/ayghri/i-have-adhd/blob/main/evals/runners.example.json)** — Template defining `claude` and `codex` runner commands
- **[`tests/test_run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/tests/test_run_evals.py)** — Test coverage for validation, planning, and scoring logic
- **[`evals/README.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/README.md)** — Human-readable usage documentation

## Summary

- **Only dependency**: Python 3.10+ (standard library only, no pip installs)
- **Optional tools**: Claude CLI or Codex CLI for model execution
- **Configuration**: [`evals/runners.json`](https://github.com/ayghri/i-have-adhd/blob/main/evals/runners.json) maps runner names to shell commands
- **Zero Python packages**: No [`requirements.txt`](https://github.com/ayghri/i-have-adhd/blob/main/requirements.txt), [`setup.py`](https://github.com/ayghri/i-have-adhd/blob/main/setup.py), or [`pyproject.toml`](https://github.com/ayghri/i-have-adhd/blob/main/pyproject.toml) dependencies

## Frequently Asked Questions

### What Python version do I need for the i-have-adhd evaluation suite?

Python 3.10 or newer. The [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) file uses `list[dict[str, Any]]` type-hint syntax standardized in PEP 585, which requires Python 3.9+, plus `pathlib` features fully stabilized in 3.10. No compatibility shims or backports are provided.

### Do I need to install any Python packages to run evaluations?

No. The evaluation suite uses only the Python standard library (`argparse`, `json`, `subprocess`, `pathlib`, `typing`, `collections`). There is no [`requirements.txt`](https://github.com/ayghri/i-have-adhd/blob/main/requirements.txt) or [`pyproject.toml`](https://github.com/ayghri/i-have-adhd/blob/main/pyproject.toml) in the repository defining external dependencies. The optional Claude and Codex runners are external CLI tools, not Python packages.

### Can I run evaluations without Claude or Codex installed?

Yes, but only partially. You can validate cases (`validate`), plan evaluation matrices (`plan`), and score results (`score`) without any external runners. However, the `run` subcommand requires a configured runner in [`evals/runners.json`](https://github.com/ayghri/i-have-adhd/blob/main/evals/runners.json). You could define a custom runner pointing to any local CLI tool that follows the expected JSONL interface.

### Where are the external runners configured?

Runner definitions live in [`evals/runners.json`](https://github.com/ayghri/i-have-adhd/blob/main/evals/runners.json) (created by copying [`evals/runners.example.json`](https://github.com/ayghri/i-have-adhd/blob/main/evals/runners.example.json)). Each entry specifies a shell command template with placeholders like `{prompt_file}` and `{output_file}` that [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) populates via `subprocess` invocation. The example file provides ready-to-use configurations for `claude` and `codex` CLIs.