Dependencies Required to Run the Evaluation Suite for 'i-have-adhd'

The evaluation suite for i-have-adhd requires only Python 3.10+ and optional external CLI runners (Claude or Codex), with no third-party Python packages needed.

The i-have-adhd repository includes a lightweight, pure-Python evaluation framework for testing LLM-assisted coding workflows. According to the source code at ayghri/i-have-adhd, the entire suite runs on the standard library and external CLI tools configured via JSON.

Core Python Requirements

The evaluation suite is intentionally minimal. Everything needed to validate, plan, run, and score evaluations is included in Python's standard library.

Python 3.10 or newer is required because scripts/run_evals.py uses modern type-hint syntax (list[dict[str, Any]]) and pathlib.Path objects that depend on recent interpreter features.

The script imports only standard library modules:

• argparse — CLI argument parsing
• json / jsonlines handling via built-in json
• subprocess — invoking external runners
• pathlib — filesystem operations

No requirements.txt or pyproject.toml dependencies exist for the evaluation suite itself.

Optional External Runners

While the Python code runs standalone, executing actual LLM evaluations requires external CLI tools defined in evals/runners.example.json:

Runner	Tool	Purpose
claude	Claude CLI	Evaluate using Anthropic's Claude model
codex	Codex CLI	Evaluate using OpenAI's Codex model

These are optional—the framework supports any runner configured in your local evals/runners.json. You only need the one matching your target model.

Installation and Verification

Clone the repository and verify your environment:

git clone https://github.com/ayghri/i-have-adhd.git
cd i-have-adhd

# Verify Python version

python3 --version  # Must be 3.10+

# Validate the evaluation suite works

python3 scripts/run_evals.py --help

External Runner Installation (Optional)

Install Claude CLI:

# Via npm (requires Node.js)

npm install -g @anthropic-ai/claude-cli

Or install Codex CLI:

# Via pip

pip install openai-codex

Then copy and customize the runner configuration:

cp evals/runners.example.json evals/runners.json

# Edit evals/runners.json to match your installed tools

Running the Evaluation Suite

Validate Case Catalog

Check that your JSONL case files are well-formed before running evaluations:

python3 scripts/run_evals.py validate

This executes the validate_cases() function in scripts/run_evals.py.

Plan Evaluation Matrix

Preview what will run without executing:

python3 scripts/run_evals.py plan \
  --trials 3 \
  --include-comparator

Outputs JSONL describing every case/trial/condition combination.

Execute Evaluation

Run with Claude as the runner:

python3 scripts/run_evals.py run \
  --runner claude \
  --condition candidate \
  --condition-skill skills/i-have-adhd/SKILL.md \
  --output evals/results/responses.jsonl

Parameters match the argparse configuration in scripts/run_evals.py:

• --runner — key from evals/runners.json
• --condition — experimental condition to test
• --condition-skill — path to skill markdown file
• --output — JSONL destination for responses

Score Results

Aggregate manually judged results:

python3 scripts/run_evals.py score evals/results/scores.jsonl

Invokes summarize_scores() to print performance metrics.

Key Source Files

Understanding the dependencies requires examining these locations:

• scripts/run_evals.py — Core CLI with validate(), plan(), run(), and score() subcommands
• evals/runners.example.json — Template defining claude and codex runner commands
• tests/test_run_evals.py — Test coverage for validation, planning, and scoring logic
• evals/README.md — Human-readable usage documentation

Summary

• Only dependency: Python 3.10+ (standard library only, no pip installs)
• Optional tools: Claude CLI or Codex CLI for model execution
• Configuration: evals/runners.json maps runner names to shell commands
• Zero Python packages: No requirements.txt, setup.py, or pyproject.toml dependencies

Frequently Asked Questions

What Python version do I need for the i-have-adhd evaluation suite?

Python 3.10 or newer. The scripts/run_evals.py file uses list[dict[str, Any]] type-hint syntax standardized in PEP 585, which requires Python 3.9+, plus pathlib features fully stabilized in 3.10. No compatibility shims or backports are provided.

Do I need to install any Python packages to run evaluations?

No. The evaluation suite uses only the Python standard library (argparse, json, subprocess, pathlib, typing, collections). There is no requirements.txt or pyproject.toml in the repository defining external dependencies. The optional Claude and Codex runners are external CLI tools, not Python packages.

Can I run evaluations without Claude or Codex installed?

Yes, but only partially. You can validate cases (validate), plan evaluation matrices (plan), and score results (score) without any external runners. However, the run subcommand requires a configured runner in evals/runners.json. You could define a custom runner pointing to any local CLI tool that follows the expected JSONL interface.

Where are the external runners configured?

Runner definitions live in evals/runners.json (created by copying evals/runners.example.json). Each entry specifies a shell command template with placeholders like {prompt_file} and {output_file} that scripts/run_evals.py populates via subprocess invocation. The example file provides ready-to-use configurations for claude and codex CLIs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}