# Limitations of the Current Evaluation Methods for *i‑have‑adhd*

> Discover the limitations of the current i-have-adhd evaluation framework. Understand how manual scoring and rigid requirements create scalability bottlenecks.

- Repository: [Ayoub Ghriss/i-have-adhd](https://github.com/ayghri/i-have-adhd)
- Tags: deep-dive
- Published: 2026-08-03

---

**The *i‑have‑adhd* evaluation framework relies on manual human scoring, strict row‑pairing requirements, and cost‑only budgeting, which together create scalability bottlenecks and reduce experimental flexibility.**

The *i‑have‑adhd* project by ayghri provides a reproducible, budget‑aware pipeline for comparing AI response styles, but its evaluation methods carry significant constraints that impact adaptability and automation. Understanding these limitations helps developers decide when the current harness suffices and when custom extensions become necessary.

## Manual Scoring Creates Subjectivity and Scale Barriers

The `summarize_scores` function in [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py) (lines 31‑68) expects human‑produced score rows with fields like `correctness`, `autonomy`, `actionability`, `safety`, `concision`, and `blocker`. No automated metric extraction exists—token‑level or latency data must be manually injected if desired.

This **human‑in‑the‑loop design** introduces two problems:

- **Subjectivity**: Different scorers may apply the rubric inconsistently
- **Throughput ceiling**: Evaluation speed depends entirely on human availability

The rubric itself ([`evals/rubric.md`](https://github.com/ayghri/i-have-adhd/blob/main/evals/rubric.md)) is static and shared across all cases. Adding case‑specific criteria requires editing the rubric file rather than per‑case configuration, reducing granularity for specialized domains.

## Strict Row Pairing Limits Experimental Flexibility

The `_check_pairing` function (lines 101‑127) enforces that every condition—baseline, candidate, and optional comparator—must be judged on *exactly* the same `(case_id, trial)` rows. Any deviation raises a hard error.

This constraint prevents:

- Evaluating a condition on a targeted subset of cases
- Adding new case types to an existing matrix without full regeneration
- Incremental experimentation with partial coverage

The same function also rejects duplicate score rows for identical `(case_id, trial, condition, runner)` tuples (lines 108‑112). While this prevents data contamination, it makes resumable runs fragile—accidental re‑submission of a completed row aborts the entire scoring step.

## Cost‑Only Budgeting Ignores Other Resource Constraints

The budget enforcement in `run_evaluations` (lines 21‑27, 35‑37, 84‑92) tracks only USD cost reported by the runner via a `cost_usd` JSON field. It does not consider:

- Token usage quotas
- API latency or timeouts
- Provider rate limits

Runs may terminate early due to cost limits while under‑utilizing allowed token quotas, or conversely hit hidden provider caps despite remaining budget. Runners lacking cost reporting are rejected unless `--allow-unmetered` is set, forcing custom wrapper scripts for many LLM APIs.

## Fixed Configuration Reduces Adaptability

Several hard‑coded elements limit customization:

| Component | Location | Constraint |
|-----------|----------|------------|
| **Rubric weights** | `WEIGHTS` dict, lines 20‑25 | Changing priorities requires code modification |
| **Runner cost field** | Runtime check, lines 84‑92 | Non‑compliant providers need wrappers |
| **Output format** | `run_evaluations`, lines 90‑99 | No schema versioning or provenance metadata |

The scoring rubric's fixed weighting scheme means emphasizing safety over concision—or any other priority shift—demands editing source code rather than adjusting a configuration file.

## Limited Test Coverage for Failure Modes

The test suite in [`tests/test_run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/tests/test_run_evals.py) validates:

- Case catalog validation
- Score aggregation and weighting
- Cost gating logic
- Duplicate row detection

However, it does **not** exercise network errors, partial runner crashes, malformed runner configurations, or other real‑world failure modes. Production runs may encounter untested edge cases leading to silent failures or incomplete data capture.

## Practical Impact on Workflow Efficiency

These limitations manifest in daily usage patterns. When re‑running a subset of trials after a failure, the duplicate‑row guard requires careful tracking of which `(case_id, trial)` combinations already exist. The strict pairing requirement means adding even one new case to an experiment necessitates regenerating and re‑judging the entire matrix.

For teams evaluating multiple skill variants, the manual scoring bottleneck compounds—each condition comparison requires fresh human judgment rather than automated metric extraction.

## Summary

The *i‑have‑adhd* evaluation framework provides solid foundations for reproducible paired comparisons, but developers should understand its constraints:

- **Manual scoring** is the sole source of final metrics, limiting scale and introducing subjectivity
- **Strict row pairing** prevents flexible subset experiments and incremental case additions
- **Cost‑only budgeting** ignores tokens, latency, and rate limits
- **Runner compatibility requirements** force wrapper scripts for many providers
- **Hard‑coded weights and static rubrics** resist configuration‑driven adaptation
- **Limited failure‑mode testing** leaves edge cases unexercised

Teams requiring automated evaluation, dynamic prioritization, or provider flexibility will need to extend or replace components of the current harness.

## Frequently Asked Questions

### Can the evaluation run fully automatically without human scoring?

No. According to the *i‑have‑adhd* source code, `summarize_scores` requires human‑entered JSON‑L score files with rubric fields including `correctness`, `autonomy`, and `safety`. No automated metric extraction exists in the current implementation.

### Why does adding new cases require regenerating the entire matrix?

The `_check_pairing` function enforces identical `(case_id, trial)` rows across all conditions. Any new case creates row mismatches that trigger validation errors, preventing targeted experiments or incremental expansion.

### What happens if a runner doesn't report cost in USD?

Unless `--allow-unmetered` is set, the runtime check in lines 84‑92 rejects runners lacking a `cost_usd` field. This requires developers to write custom wrapper scripts for APIs that don't provide this metadata natively.

### How can I change the priority of different scoring criteria?

Modify the `WEIGHTS` dictionary defined in lines 20‑25 of [`scripts/run_evals.py`](https://github.com/ayghri/i-have-adhd/blob/main/scripts/run_evals.py). The current design has no configuration file for weight adjustments—any prioritization change requires editing source code and redeploying the harness.