How Frequently Are the Evaluations for 'i-have-adhd' Updated?

Evaluations for the i-have-adhd repository are updated only when maintainers manually trigger them via scripts/run_evals.py—there is no automated schedule, cron job, or CI pipeline that refreshes them.

The i-have-adhd project includes a complete evaluation harness in the evals/ directory for benchmarking AI-assisted ADHD coaching responses. Understanding how often these evaluations update helps contributors and researchers know when test results reflect the latest code changes. According to the repository source code, evaluation updates follow a manual, intention-driven workflow rather than any fixed cadence.

Manual Triggering via run_evals.py

All evaluation operations flow through scripts/run_evals.py, a command-line tool that handles validation, planning, execution, and scoring. The script is not invoked automatically—maintainers run it explicitly when they want fresh results.

The script supports four primary commands:

# Validate the evaluation definition

python3 scripts/run_evals.py validate

# Generate an execution plan

python3 scripts/run_evals.py plan --trials 3 --include-comparator

# Execute a full evaluation run

python3 scripts/run_evals.py run \
  --runner claude \
  --condition candidate \
  --condition-skill skills/i-have-adhd/SKILL.md \
  --trials 3 \
  --budget-usd 12.50 \
  --output evals/results/responses.jsonl

# Score the collected responses

python3 scripts/run_evals.py score evals/results/scores.jsonl

Because each command requires explicit invocation, the frequency of evaluation updates depends entirely on maintainer discretion.

What Triggers an Evaluation Update

Updates typically occur in these scenarios:

• After code changes to the skill definition in skills/i-have-adhd/SKILL.md
• When benchmarking a new model version or different runner (e.g., switching from Claude to another provider)
• Prior to releases to validate performance against the rubric
• Ad hoc comparisons between baseline and candidate conditions

There is no evidence in the repository of GitHub Actions workflows, cron schedules, or other automation that would trigger run_evals.py without human intervention.

Static Evaluation Assets

The underlying test cases and scoring criteria remain stable unless manually edited. Key files include:

• evals/cases.jsonl — JSONL file containing all evaluation cases (test scenarios)
• evals/rubric.md — Scoring rubric defining dimensions, weights, and blocking criteria
• evals/README.md — Documentation for the evaluation workflow

These files are version-controlled but not regenerated automatically. Only the results produced by run_evals.py change with each manual execution.

Complete Evaluation Workflow Example

To perform a full baseline-versus-candidate comparison that updates stored results:

# 1. Validate definitions

python3 scripts/run_evals.py validate

# 2. Run both conditions

python3 scripts/run_evals.py run \
  --runner claude \
  --condition baseline \
  --condition candidate \
  --condition-skill skills/i-have-adhd/SKILL.md \
  --trials 3 \
  --budget-usd 12.50 \
  --output evals/results/responses.jsonl

# 3. Generate scores

python3 scripts/run_evals.py score evals/results/scores.jsonl

Each step requires deliberate execution. The resulting responses.jsonl and score files represent the latest intentional evaluation—not a continuously updated dataset.

Summary

• No automation: Evaluations lack cron jobs, CI triggers, or scheduled pipelines
• Manual execution: scripts/run_evals.py must be invoked explicitly by maintainers
• Trigger events: Code changes, new skill versions, or model benchmarking needs
• Static inputs: evals/cases.jsonl, evals/rubric.md, and related files change only through commits
• Result pacing: Update frequency equals how often maintainers choose to run evaluations

Frequently Asked Questions

Is there a daily or weekly schedule for running i-have-adhd evaluations?

No. The repository contains no automated scheduling mechanism. Evaluations update only when someone manually executes scripts/run_evals.py.

How can I check when the most recent evaluation was performed?

Examine the commit history of evals/results/ files or any timestamped outputs from previous run_evals.py executions. The repository does not expose a dedicated "last run" endpoint.

Can I trigger my own evaluation run as an external contributor?

Yes. Clone the repository, ensure you have API credentials for your chosen runner (e.g., Claude), and execute the run_evals.py commands locally with your own budget parameters.

What does it cost to run a full evaluation suite?

The example commands specify --budget-usd 12.50, suggesting a typical run costs approximately $10–15 depending on trial count and model pricing. Adjust --budget-usd and --trials to control expenses.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}