How i-have-adhd Addresses Evaluation Data Bias: A Technical Deep-Dive

The i-have-adhd project implements a seven-layer bias mitigation strategy including blind judging, weighted scoring, diverse risk-tagged cases, strict validation, paired-condition checks, release gates, and runner isolation to ensure fair model comparisons.

Bias in evaluation data can sink even the most promising AI system comparisons. The i-have-adhd repository, which measures whether injecting an ADHD-aware response style improves Claude or Codex outputs without harming correctness, treats unbiased evaluation as a first-class engineering concern. Every component of its evaluation pipeline—from case selection to final release decisions—is designed to prevent systematic skew that could favor either the baseline or the candidate condition.

Blind Judging: Eliminating Evaluator Bias

Human evaluators, even well-intentioned ones, can be influenced by knowing which model produced a response. The i-have-adhd evaluation requires anonymous labeling.

The rubric in evals/rubric.md explicitly instructs judges to "label them A, B, or C without exposing the condition name" (lines 3-5). This simple but critical rule prevents halo effects, confirmation bias, and brand-based assumptions from contaminating scores.

Weighted Scoring: Prioritizing What Matters

Not all evaluation dimensions deserve equal influence. The project assigns higher weights to correctness and safety to reduce noise from less critical differences.

The WEIGHTS constant in scripts/run_evals.py (lines 17-25) codifies these priorities, and the same weights appear in the rubric's scoring table. This prevents a candidate from "gaming" the evaluation by excelling on stylistic dimensions while regressing on substance.

Diverse, Risk-Tagged Case Selection

A narrow case set risks overfitting to specific scenarios. The evals/cases.jsonl file contains 14 distinct cases spanning coding, safety, medical queries, ambiguous requests, and more (lines 1-15).

Each case carries an explicit risk field (low, medium, or high), ensuring the evaluation stress-tests the candidate across severity tiers rather than cherry-picking easy wins.

Strict Validation: Catching Hidden Skew Early

Invalid data silently corrupts results. The validate_cases function in scripts/run_evals.py (lines 59-77) enforces:

• Required keys — no missing fields
• Unique IDs — no duplicate cases inflating scores
• Valid risk values — only low/medium/high allowed

This validation runs automatically before any evaluation begins, failing fast rather than propagating errors into final metrics.

Paired-Condition Pairing Checks

Fair comparison demands identical inputs across conditions. The _check_pairing function (lines 101-127) raises an error if:

• Any condition is missing rows present in the baseline
• Any condition has extra rows not in the baseline

This eliminates selection bias—the candidate cannot be evaluated on an easier subset while the baseline faces harder cases.

Release Gate: Objective Pass/Fail Criteria

Subjective "looks better" judgments invite bias. The summarize_scores function (lines 155-162) enforces hard thresholds:

Criterion	Threshold
Correctness regression	≤ 0.1 point drop
Safety regression	≤ 0.1 point drop
Overall result	Weighted score must beat baseline

A candidate must clear all three hurdles to pass. No manual overrides, no "good enough" exceptions.

Runner Configuration Isolation

Environment leakage can unfairly advantage one condition. The evaluation enforces fresh, empty settings for each run:

# Claude: explicit empty setting sources

--setting-sources ""

# Codex: ignore user config, use ephemeral context

--ignore-user-config --ephemeral

The README (lines 35-38) specifically warns against the "always-on" flag that would inject the skill into the baseline, destroying the controlled comparison.

Running the Full Evaluation Pipeline

Execute the complete bias-aware workflow:

# 1. Validate case catalog

python3 scripts/run_evals.py validate

# 2. Generate blind-judgment run matrix

python3 scripts/run_evals.py plan --trials 3 --include-comparator > matrix.jsonl

# 3. Run baseline (no skill)

python3 scripts/run_evals.py run \
  --runner claude \
  --condition baseline \
  --trials 3 \
  --budget-usd 12.5 \
  --output evals/results/baseline.jsonl

# 4. Run candidate (with ADHD skill)

python3 scripts/run_evals.py run \
  --runner claude \
  --condition candidate \
  --condition-skill skills/i-have-adhd/SKILL.md \
  --trials 3 \
  --budget-usd 12.5 \
  --output evals/results/candidate.jsonl

# 5. Score blind judgments with weighted aggregation

python3 scripts/run_evals.py score evals/results/scores.jsonl

Summary

• Blind judging prevents evaluator awareness from skewing scores
• Weighted dimensions prioritize correctness and safety over noise
• 14 risk-tagged cases ensure diverse, representative evaluation coverage
• Strict validation catches malformed data before it corrupts results
• Paired-condition checks guarantee identical inputs across conditions
• Release gates enforce objective, non-negotiable pass criteria
• Runner isolation eliminates configuration contamination

Frequently Asked Questions

What is blind judging in i-have-adhd evaluation?

Blind judging means evaluators score responses labeled A, B, or C without knowing which condition (baseline or candidate) produced each. This prevents unconscious bias from affecting scores. The rubric in evals/rubric.md mandates this anonymity.

How does weighted scoring reduce bias in i-have-adhd?

Weighted scoring assigns higher importance to correctness and safety dimensions compared to stylistic factors. This prevents a candidate from appearing superior by excelling on less critical criteria while actually regressing on substantive quality. The weights are defined in scripts/run_evals.py and enforced in summarize_scores.

What happens if case data fails validation?

The validate_cases function rejects the evaluation entirely, printing specific errors about duplicate IDs, missing fields, or invalid risk values. This fail-fast approach prevents corrupted data from producing misleading benchmark results.

Why is runner isolation important for fair evaluation?

Runner isolation ensures each condition starts with a clean configuration, preventing skill files or user preferences from leaking into the baseline run. Without flags like --setting-sources "" (Claude) or --ignore-user-config (Codex), the "baseline" would secretly include the skill being tested, invalidating the comparison.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}