Limitations of the Current Evaluation Methods for *i‑have‑adhd*

The i‑have‑adhd evaluation framework relies on manual human scoring, strict row‑pairing requirements, and cost‑only budgeting, which together create scalability bottlenecks and reduce experimental flexibility.

The i‑have‑adhd project by ayghri provides a reproducible, budget‑aware pipeline for comparing AI response styles, but its evaluation methods carry significant constraints that impact adaptability and automation. Understanding these limitations helps developers decide when the current harness suffices and when custom extensions become necessary.

Manual Scoring Creates Subjectivity and Scale Barriers

The summarize_scores function in scripts/run_evals.py (lines 31‑68) expects human‑produced score rows with fields like correctness, autonomy, actionability, safety, concision, and blocker. No automated metric extraction exists—token‑level or latency data must be manually injected if desired.

This human‑in‑the‑loop design introduces two problems:

• Subjectivity: Different scorers may apply the rubric inconsistently
• Throughput ceiling: Evaluation speed depends entirely on human availability

The rubric itself (evals/rubric.md) is static and shared across all cases. Adding case‑specific criteria requires editing the rubric file rather than per‑case configuration, reducing granularity for specialized domains.

Strict Row Pairing Limits Experimental Flexibility

The _check_pairing function (lines 101‑127) enforces that every condition—baseline, candidate, and optional comparator—must be judged on exactly the same (case_id, trial) rows. Any deviation raises a hard error.

This constraint prevents:

• Evaluating a condition on a targeted subset of cases
• Adding new case types to an existing matrix without full regeneration
• Incremental experimentation with partial coverage

The same function also rejects duplicate score rows for identical (case_id, trial, condition, runner) tuples (lines 108‑112). While this prevents data contamination, it makes resumable runs fragile—accidental re‑submission of a completed row aborts the entire scoring step.

Cost‑Only Budgeting Ignores Other Resource Constraints

The budget enforcement in run_evaluations (lines 21‑27, 35‑37, 84‑92) tracks only USD cost reported by the runner via a cost_usd JSON field. It does not consider:

• Token usage quotas
• API latency or timeouts
• Provider rate limits

Runs may terminate early due to cost limits while under‑utilizing allowed token quotas, or conversely hit hidden provider caps despite remaining budget. Runners lacking cost reporting are rejected unless --allow-unmetered is set, forcing custom wrapper scripts for many LLM APIs.

Fixed Configuration Reduces Adaptability

Several hard‑coded elements limit customization:

Component	Location	Constraint
Rubric weights	WEIGHTS dict, lines 20‑25	Changing priorities requires code modification
Runner cost field	Runtime check, lines 84‑92	Non‑compliant providers need wrappers
Output format	run_evaluations, lines 90‑99	No schema versioning or provenance metadata

The scoring rubric's fixed weighting scheme means emphasizing safety over concision—or any other priority shift—demands editing source code rather than adjusting a configuration file.

Limited Test Coverage for Failure Modes

The test suite in tests/test_run_evals.py validates:

• Case catalog validation
• Score aggregation and weighting
• Cost gating logic
• Duplicate row detection

However, it does not exercise network errors, partial runner crashes, malformed runner configurations, or other real‑world failure modes. Production runs may encounter untested edge cases leading to silent failures or incomplete data capture.

Practical Impact on Workflow Efficiency

These limitations manifest in daily usage patterns. When re‑running a subset of trials after a failure, the duplicate‑row guard requires careful tracking of which (case_id, trial) combinations already exist. The strict pairing requirement means adding even one new case to an experiment necessitates regenerating and re‑judging the entire matrix.

For teams evaluating multiple skill variants, the manual scoring bottleneck compounds—each condition comparison requires fresh human judgment rather than automated metric extraction.

Summary

The i‑have‑adhd evaluation framework provides solid foundations for reproducible paired comparisons, but developers should understand its constraints:

• Manual scoring is the sole source of final metrics, limiting scale and introducing subjectivity
• Strict row pairing prevents flexible subset experiments and incremental case additions
• Cost‑only budgeting ignores tokens, latency, and rate limits
• Runner compatibility requirements force wrapper scripts for many providers
• Hard‑coded weights and static rubrics resist configuration‑driven adaptation
• Limited failure‑mode testing leaves edge cases unexercised

Teams requiring automated evaluation, dynamic prioritization, or provider flexibility will need to extend or replace components of the current harness.

Frequently Asked Questions

Can the evaluation run fully automatically without human scoring?

No. According to the i‑have‑adhd source code, summarize_scores requires human‑entered JSON‑L score files with rubric fields including correctness, autonomy, and safety. No automated metric extraction exists in the current implementation.

Why does adding new cases require regenerating the entire matrix?

The _check_pairing function enforces identical (case_id, trial) rows across all conditions. Any new case creates row mismatches that trigger validation errors, preventing targeted experiments or incremental expansion.

What happens if a runner doesn't report cost in USD?

Unless --allow-unmetered is set, the runtime check in lines 84‑92 rejects runners lacking a cost_usd field. This requires developers to write custom wrapper scripts for APIs that don't provide this metadata natively.

How can I change the priority of different scoring criteria?

Modify the WEIGHTS dictionary defined in lines 20‑25 of scripts/run_evals.py. The current design has no configuration file for weight adjustments—any prioritization change requires editing source code and redeploying the harness.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}