How Edge Cases Are Handled and Tested in the *i-have-adhd* Repository

The i-have-adhd project handles edge cases through robust input validation, model-agnostic fallback logic, and a comprehensive test suite in tests/test_run_evals.py that deliberately exercises malformed inputs, API failures, and configuration errors.

This open-source evaluation runner processes LLM test cases via a command-line interface in scripts/run_evals.py. The codebase demonstrates defensive programming patterns designed to survive real-world deployment conditions—including bad data, missing credentials, and network instability. This article examines exactly how edge cases are managed and verified according to the ayghri/i-have-adhd source code.

---

Input Validation and Malformed Data Handling

The runner's first line of defense operates in scripts/run_evals.py, where every entry from evals/cases.jsonl undergoes strict validation.

Required Field Verification

Each case must contain prompt, expected, and model keys. The validate_case() function raises an explicit ValueError when any required field is absent, preventing downstream failures from cryptic stack traces.

# From tests/test_run_evals.py — validates missing field detection

def test_missing_prompt_raises():
    malformed_case = {"expected": "Yes", "model": "openai"}  # `prompt` omitted

    with pytest.raises(ValueError, match="Missing required field 'prompt'"):
        validate_case(malformed_case)

Input Size and Format Normalization

The runner implements two additional protective layers:

• Token limit enforcement — Prompts exceeding model context windows are truncated or padded before API submission
• Whitespace normalization — OS-specific line endings (\r\n vs \n) and surrounding whitespace are standardized so platform differences don't cause false negatives

These transformations occur before any external API call, ensuring consistent behavior across operating systems.

---

Model Provider Flexibility and Graceful Degradation

The architecture supports dual backend providers through configuration files in skills/i-have-adhd/agents/:

Provider	Configuration File
OpenAI	agents/openai.yaml
Gemini	agents/gemini.toml

When scripts/run_evals.py detects that no API keys are configured for either provider, it falls back to a no-op stub rather than crashing. This behavior emits a clear warning and allows the evaluation batch to continue—a critical edge case for CI environments or offline development.

# Run with fail-fast disabled to continue through provider unavailability

python scripts/run_evals.py \
    --cases evals/cases.jsonl \
    --rubric evals/rubric.md

# Programmatic invocation with explicit error tolerance

from scripts.run_evals import run_evals

run_evals(
    cases_path="evals/cases.jsonl",
    rubric_path="evals/rubric.md",
    model="openai",          # or "gemini"

    fail_fast=False         # continue after errors

)

---

Result Verification with Fuzzy Matching

Post-execution validation in scripts/run_evals.py applies the scoring rubric defined in evals/rubric.md. The comparison logic intentionally tolerates:

• Case variations ("Yes" vs "yes")
• Extraneous punctuation
• Leading/trailing whitespace

These are treated as acceptable variations unless the rubric explicitly flags them as critical distinctions. This prevents brittle exact-string matching from producing false failures on semantically equivalent outputs.

---

Network and API Error Recovery

All external calls to LLM providers are wrapped in comprehensive try/except blocks capturing:

• Network timeouts
• Connection failures
• API-level exceptions (rate limits, invalid requests, server errors)

Rather than aborting the entire batch, the runner logs a concise error message and records the specific test as "failed," preserving partial results and execution continuity.

---

Comprehensive Edge Case Test Coverage

The test suite in tests/test_run_evals.py systematically validates every defensive code path through deliberately malformed inputs:

Test Category	Input Condition	Expected Behavior
Missing fields	Case without prompt key	ValueError with clear message
Oversized input	Prompt exceeding token limit	Silent truncation, no exception
Invalid JSON	Malformed line in cases.jsonl	Skip line with logged warning
Unavailable model	No API keys in environment	Warning emission, continue execution

These tests execute automatically via the CI workflow defined in .github/workflows/plugin-load-check.yml, ensuring regressions in edge-case handling are caught before release.

---

Summary

• Strict validation of required fields (prompt, expected, model) with explicit ValueError exceptions
• Input normalization for token limits and cross-platform whitespace consistency
• Provider-agnostic fallback to no-op stubs when API credentials are unavailable
• Fuzzy result matching against evals/rubric.md with configurable strictness
• Graceful degradation on network failures—log and continue rather than crash
• Systematic test coverage in tests/test_run_evals.py exercising all failure modes

---

Frequently Asked Questions

What happens if a test case is missing the required prompt field?

The validate_case() function in scripts/run_evals.py raises a ValueError with the message "Missing required field 'prompt'". This is verified by test_missing_prompt_raises() in the test suite.

How does the runner handle prompts that exceed the model's context window?

Input strings are automatically truncated or padded to fit within token limits before API submission. The test_oversized_input case confirms this occurs without raising exceptions.

Can the evaluation suite run without API credentials configured?

Yes. When neither OpenAI nor Gemini credentials are present, the runner falls back to a no-op stub, emits a warning, and continues processing. This is tested via environment simulation in tests/test_run_evals.py.

Where are the edge-case tests executed in CI?

All tests run through the workflow defined in .github/workflows/plugin-load-check.yml, which executes tests/test_run_evals.py against the full matrix of malformed inputs and failure conditions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}