How to Contribute to Testing the `i-have-adhd` Skill: A Complete Guide

Yes, you can contribute to testing the i-have-adhd skill by adding evaluation cases to evals/cases.jsonl, creating score fixtures, running the local test suite, and submitting a pull request.

The i-have-adhd skill is a response-style plug-in that reshapes LLM output to better serve readers with ADHD. According to the ayghri/i-have-adhd source code, the testing framework is explicitly designed for community contributions, with clear extension points in evals/cases.jsonl and comprehensive unit tests in tests/test_run_evals.py.

Understanding the Test Architecture Before You Contribute

The testing framework splits responsibilities across four key components. Understanding these will help you target your contributions effectively.

Component	Role	Key Source
Skill definition	YAML-style rules file read by the harness and injected into prompts when the condition is candidate	skills/i-have-adhd/SKILL.md
Evaluation cases	JSON-Lines catalog of prompts, risk levels, and success criteria	evals/cases.jsonl
Test harness	Loads cases, builds prompts, invokes runners, parses responses, aggregates scores	scripts/run_evals.py
Unit tests	Validates harness behavior: validation, pairing, scoring, budget logic	tests/test_run_evals.py

How the Harness Orchestrates Test Runs

The run_evals.py harness operates in three sequential stages:

1. Validate the case catalog — validate_cases() (lines 59-79) enforces required fields (id, category, prompt, risk, criteria) and checks for duplicate IDs
2. Run evaluations — run_evaluations() (lines 9-100) iterates over cases, builds prompts via _condition_prompt() (lines 71-82) inserting skill text when condition is candidate, then calls the configured runner
3. Score results — summarize_scores() (lines 30-68) applies weighted metrics and release-gate rules to determine if the candidate passes

Four Ways to Contribute to i-have-adhd Skill Testing

1. Add New Evaluation Cases

The simplest entry point: extend evals/cases.jsonl with additional test scenarios.

Each case must follow this schema:

{"id":"simple-math","category":"direct-answer","prompt":"What is 7 × 8?","risk":"low","criteria":["Answers 56."]}

Required fields:

• id — unique identifier (enforced by validation)
• category — classification for grouping results
• prompt — the actual input sent to the LLM
• risk — typically low, medium, or high
• criteria — array of strings defining success conditions

To add a case programmatically:

import json, pathlib

new_case = {
    "id": "simple-math",
    "category": "direct-answer",
    "prompt": "What is 7 × 8?",
    "risk": "low",
    "criteria": ["Answers 56."]
}
cases_path = pathlib.Path("evals/cases.jsonl")
with cases_path.open("a", encoding="utf-8") as f:
    f.write(json.dumps(new_case, ensure_ascii=False) + "\n")

The unit test test_case_catalog_is_valid_and_balanced automatically verifies your addition.

2. Create Score Fixtures for Direct Scoring Tests

To test scoring logic without running live LLM calls, create .jsonl files mimicking runner output.

Each row must contain fields validated by _validate_score() (lines 82-95 in run_evals.py). Use these fixtures to verify edge cases in the scoring pipeline.

3. Run and Verify Your Changes Locally

Execute the full test suite before submitting:

python -m unittest discover -s tests

This validates:

• Case catalog integrity
• Field pairing logic
• Duplicate ID detection
• Budget handling
• Score aggregation accuracy

4. Run the Evaluation Harness Manually

Test your cases with actual skill injection:

python scripts/run_evals.py run \
  --runner-config evals/runners.example.json \
  --runner stub \
  --condition candidate \
  --condition-skill skills/i-have-adhd/SKILL.md \
  --output evals/out.jsonl

Then verify scoring:

python scripts/run_evals.py score evals/out.jsonl

This outputs weighted scores and the release-gate pass/fail decision.

Submitting Your i-have-adhd Testing Contribution

1. Fork the ayghri/i-have-adhd repository
2. Branch from main with a descriptive name (add-math-evaluation-cases)
3. Commit your changes to evals/cases.jsonl and any new fixtures
4. Push and open a pull request

The CI pipeline (.github/workflows/plugin-load-check.yml) automatically runs the same test suite, blocking merges that break existing behavior.

Key Files for Contributors

Path	Purpose
skills/i-have-adhd/SKILL.md	10 output-style rules defining the skill's behavior
evals/cases.jsonl	Extend this file with new test cases
scripts/run_evals.py	Core harness with validate_cases(), run_evaluations(), summarize_scores()
tests/test_run_evals.py	Unit tests ensuring harness reliability
evals/runners.example.json	Template runner configuration

Summary

• Yes, contributions are explicitly supported — the framework is designed for community extension
• Add cases to evals/cases.jsonl — validated automatically by validate_cases() in run_evals.py
• Run python -m unittest discover -s tests — local verification before PR submission
• Use --condition candidate — to test skill injection via _condition_prompt()
• CI enforces quality gates — matching local test behavior exactly

Frequently Asked Questions

What programming skills do I need to contribute to i-have-adhd skill testing?

Basic Python and JSON familiarity suffice. Adding evaluation cases requires only editing evals/cases.jsonl. For deeper contributions, understanding the harness functions in run_evals.py (particularly _condition_prompt() and _validate_score()) helps, but the existing unit tests provide guardrails.

How does the harness inject the ADHD skill into test prompts?

The _condition_prompt() function (lines 71-82) conditionally prepends skill content from SKILL.md when the --condition flag is set to candidate. This lets you compare baseline LLM responses against skill-modified output using identical evaluation cases.

Can I test my cases without calling a live LLM?

Yes. The --runner stub option uses a mock runner. You can also create score fixtures and run python scripts/run_evals.py score directly against saved outputs, bypassing LLM calls entirely while still validating scoring logic.

What makes a good evaluation case for the i-have-adhd skill?

Focus on prompts where ADHD-optimized formatting genuinely matters: long explanations, multi-step instructions, or dense technical content. The risk field should reflect real-world impact, and criteria must be objectively verifiable (e.g., "Uses bullet points" not "Is easy to read").

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}