# How i-have-adhd Addresses Evaluation Data Bias: A Technical Deep-Dive

> Discover how i-have-adhd tackles evaluation data bias with a seven-layer strategy including blind judging and weighted scoring for fair model comparisons.

- Repository: [Ayoub Ghriss/i-have-adhd](https://github.com/ayghri/i-have-adhd)
- Tags: deep-dive
- Published: 2026-08-03

---

**The i-have-adhd project implements a seven-layer bias mitigation strategy including blind judging, weighted scoring, diverse risk-tagged cases, strict validation, paired-condition checks, release gates, and runner isolation to ensure fair model comparisons.**

Bias in evaluation data can sink even the most promising AI system comparisons. The i-have-adhd repository, which measures whether injecting an ADHD-aware response style improves Claude or Codex outputs without harming correctness, treats unbiased evaluation as a first-class engineering concern. Every component of its evaluation pipeline—from case selection to final release decisions—is designed to prevent systematic skew that could favor either the baseline or the candidate condition.

## Blind Judging: Eliminating Evaluator Bias

Human evaluators, even well-intentioned ones, can be influenced by knowing which model produced a response. The i-have-adhd evaluation **requires anonymous labeling**.

The rubric in [`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md) explicitly instructs judges to "label them `A`, `B`, or `C` without exposing the condition name" (lines 3-5). This simple but critical rule prevents halo effects, confirmation bias, and brand-based assumptions from contaminating scores.

## Weighted Scoring: Prioritizing What Matters

Not all evaluation dimensions deserve equal influence. The project assigns **higher weights to correctness and safety** to reduce noise from less critical differences.

The `WEIGHTS` constant in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) (lines 17-25) codifies these priorities, and the same weights appear in the rubric's scoring table. This prevents a candidate from "gaming" the evaluation by excelling on stylistic dimensions while regressing on substance.

## Diverse, Risk-Tagged Case Selection

A narrow case set risks overfitting to specific scenarios. The `evals/cases.jsonl` file contains **14 distinct cases** spanning coding, safety, medical queries, ambiguous requests, and more (lines 1-15).

Each case carries an explicit `risk` field (low, medium, or high), ensuring the evaluation stress-tests the candidate across severity tiers rather than cherry-picking easy wins.

## Strict Validation: Catching Hidden Skew Early

Invalid data silently corrupts results. The `validate_cases` function in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) (lines 59-77) enforces:

- **Required keys** — no missing fields
- **Unique IDs** — no duplicate cases inflating scores
- **Valid risk values** — only low/medium/high allowed

This validation runs automatically before any evaluation begins, failing fast rather than propagating errors into final metrics.

## Paired-Condition Pairing Checks

Fair comparison demands **identical inputs across conditions**. The `_check_pairing` function (lines 101-127) raises an error if:

- Any condition is missing rows present in the baseline
- Any condition has extra rows not in the baseline

This eliminates selection bias—the candidate cannot be evaluated on an easier subset while the baseline faces harder cases.

## Release Gate: Objective Pass/Fail Criteria

Subjective "looks better" judgments invite bias. The `summarize_scores` function (lines 155-162) enforces hard thresholds:

| Criterion | Threshold |
|-----------|-----------|
| Correctness regression | ≤ 0.1 point drop |
| Safety regression | ≤ 0.1 point drop |
| Overall result | Weighted score must beat baseline |

A candidate must clear all three hurdles to pass. No manual overrides, no "good enough" exceptions.

## Runner Configuration Isolation

Environment leakage can unfairly advantage one condition. The evaluation enforces **fresh, empty settings** for each run:

```bash

# Claude: explicit empty setting sources

--setting-sources ""

# Codex: ignore user config, use ephemeral context

--ignore-user-config --ephemeral

```

The README (lines 35-38) specifically warns against the "always-on" flag that would inject the skill into the baseline, destroying the controlled comparison.

## Running the Full Evaluation Pipeline

Execute the complete bias-aware workflow:

```bash

# 1. Validate case catalog

python3 scripts/run_evals.py validate

# 2. Generate blind-judgment run matrix

python3 scripts/run_evals.py plan --trials 3 --include-comparator > matrix.jsonl

# 3. Run baseline (no skill)

python3 scripts/run_evals.py run \
  --runner claude \
  --condition baseline \
  --trials 3 \
  --budget-usd 12.5 \
  --output evals/results/baseline.jsonl

# 4. Run candidate (with ADHD skill)

python3 scripts/run_evals.py run \
  --runner claude \
  --condition candidate \
  --condition-skill skills/i-have-adhd/SKILL.md \
  --trials 3 \
  --budget-usd 12.5 \
  --output evals/results/candidate.jsonl

# 5. Score blind judgments with weighted aggregation

python3 scripts/run_evals.py score evals/results/scores.jsonl

```

## Summary

- **Blind judging** prevents evaluator awareness from skewing scores
- **Weighted dimensions** prioritize correctness and safety over noise
- **14 risk-tagged cases** ensure diverse, representative evaluation coverage
- **Strict validation** catches malformed data before it corrupts results
- **Paired-condition checks** guarantee identical inputs across conditions
- **Release gates** enforce objective, non-negotiable pass criteria
- **Runner isolation** eliminates configuration contamination

## Frequently Asked Questions

### What is blind judging in i-have-adhd evaluation?

Blind judging means evaluators score responses labeled `A`, `B`, or `C` without knowing which condition (baseline or candidate) produced each. This prevents unconscious bias from affecting scores. The rubric in [`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md) mandates this anonymity.

### How does weighted scoring reduce bias in i-have-adhd?

Weighted scoring assigns higher importance to correctness and safety dimensions compared to stylistic factors. This prevents a candidate from appearing superior by excelling on less critical criteria while actually regressing on substantive quality. The weights are defined in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) and enforced in `summarize_scores`.

### What happens if case data fails validation?

The `validate_cases` function rejects the evaluation entirely, printing specific errors about duplicate IDs, missing fields, or invalid risk values. This fail-fast approach prevents corrupted data from producing misleading benchmark results.

### Why is runner isolation important for fair evaluation?

Runner isolation ensures each condition starts with a clean configuration, preventing skill files or user preferences from leaking into the baseline run. Without flags like `--setting-sources ""` (Claude) or `--ignore-user-config` (Codex), the "baseline" would secretly include the skill being tested, invalidating the comparison.