How to Debug When i-have-adhd Skill Output Is Not Actionable

The fastest way to debug unactionable output from the i-have-adhd skill is to run the evaluation harness in scripts/run_evals.py to isolate prompt template mismatches, verify JSON schema compliance in skills/i-have-adhd/SKILL.md, and confirm API credentials in the agent configuration files.

The i-have-adhd skill is a Cursor-compatible agent designed to generate structured, actionable ADHD management suggestions. When the skill returns vague responses like "Take a break" instead of concrete steps, developers need a systematic approach to debug when i-have-adhd skill output is not actionable by examining the prompt template, LLM configuration, and evaluation test cases.

Common Causes of Unactionable Output

Unactionable responses typically stem from four configuration failures:

• Prompt-template mismatch – The prompt in skills/i-have-adhd/SKILL.md was edited without updating the required JSON schema in the outputs block.
• Wrong LLM provider or model – The agent config in skills/i-have-adhd/agents/openai.yaml or gemini.toml points to a model that does not support the expected response format.
• Missing or malformed API credentials – Absent OPENAI_API_KEY or GEMINI_API_KEY environment variables cause the skill to fall back to generic placeholder responses.
• Evaluation data out-of-date – The test cases in evals/cases.jsonl no longer reflect the current schema, causing false-positive passes while the live skill fails.

Step-by-Step Debugging Procedure

1. Run the Built-in Evaluation Script

Execute the evaluation harness to reproduce the failure in a controlled environment:

python scripts/run_evals.py --cases evals/cases.jsonl

The script prints a summary and, for every failing case, the exact LLM response. If failures match the "unactionable" symptom, the problem is reproducible in the test harness.

2. Inspect the Raw LLM Response

The evaluation harness writes raw JSON to stdout or evals/runners.example.json. Look for:

• Missing keys (e.g., action, details)
• Extra newlines or stray text before the JSON block
• Truncated output caused by token limits

3. Validate the Prompt Template

Open skills/i-have-adhd/SKILL.md and verify that the system and user sections explicitly request a JSON response matching the schema defined in the skill's outputs block. The template should include:

### Output format

```json
{
  "action": "<short command>",
  "details": "<optional longer explanation>"
}

If the template is missing the `json` code fence or the schema has changed, update it and re-run the evaluation.

### 4. Check the Agent Configuration

- **For OpenAI**: Ensure [`skills/i-have-adhd/agents/openai.yaml`](https://github.com/ayghri/i-have-adhd/blob/main/skills/i-have-adhd/agents/openai.yaml) points to a model supporting the `json_object` response format (e.g., `gpt-4o-mini`).
- **For Gemini**: Confirm [`skills/i-have-adhd/agents/gemini.toml`](https://github.com/ayghri/i-have-adhd/blob/main/skills/i-have-adhd/agents/gemini.toml) includes `response_format = "json"` or uses Gemini's function calling feature.

Misconfigured models often return free-form text instead of required JSON.

### 5. Verify API Credentials

The skill expects environment variables `OPENAI_API_KEY` or `GEMINI_API_KEY`. If absent, the LLM client raises an authentication error and the skill falls back to a placeholder. Create a `.env` file if missing:

```text
OPENAI_API_KEY=your-openai-key-here
GEMINI_API_KEY=your-gemini-key-here

6. Synchronize the Test Cases

Open evals/cases.jsonl and confirm each case's expected_output matches the current schema. Adjust outdated expectations and re-run the evaluation.

7. Iterate and Validate

After each change, re-run python scripts/run_evals.py until all cases pass. Then manually invoke the skill via the Cursor UI to confirm live output is actionable.

Quick Diagnostic Checklist

• python scripts/run_evals.py → all tests pass?
• skills/i-have-adhd/SKILL.md → JSON fence present and schema matches outputs block?
• Agent file (openai.yaml / gemini.toml) → model supports JSON output?
• .env → valid API keys available?
• evals/cases.jsonl → expected output aligns with current schema?

If any box remains unchecked, that component is the most likely source of unactionable output.

Code Examples and Expected Output

Run the evaluation suite manually:

# Run full evaluation suite

python scripts/run_evals.py --cases evals/cases.jsonl

# Manual skill invocation via Cursor CLI

cursor run skills/i-have-adhd --input "I feel overwhelmed by paperwork"

Sample failing output (captured by the evaluator):

{
  "action": "Take a break",
  "details": ""
}

After fixing the prompt to enforce required fields:

{
  "action": "Schedule a 10-minute Pomodoro",
  "details": "Set a timer for 10 minutes, work on a single task, then rest for 2 minutes."
}

Summary

• Run scripts/run_evals.py to isolate whether unactionable output stems from prompt, model, or credential issues.
• Verify skills/i-have-adhd/SKILL.md contains a properly fenced JSON schema that matches the outputs block.
• Confirm agent configurations in skills/i-have-adhd/agents/openai.yaml and gemini.toml use models supporting structured JSON output.
• Ensure OPENAI_API_KEY or GEMINI_API_KEY environment variables are set to prevent fallback placeholder responses.
• Keep evals/cases.jsonl synchronized with the current schema to avoid false positives in testing.

Frequently Asked Questions

Why does the i-have-adhd skill return "Take a break" instead of specific actions?

This occurs when the prompt template in SKILL.md lacks explicit JSON schema constraints or when the LLM model does not support structured output formats. The model defaults to generic advice rather than the structured action and details fields. Verify the json code fence is present in the prompt and the agent configuration specifies a JSON-compatible model.

How do I know if my API credentials are causing unactionable output?

If OPENAI_API_KEY or GEMINI_API_KEY is missing or invalid, the skill cannot reach the LLM and falls back to a static placeholder response. Check for authentication errors in the evaluation script output or set the environment variables in a .env file and re-run scripts/run_evals.py to confirm connectivity.

What should I do if the evaluation passes but the live skill still fails?

This indicates the test cases in evals/cases.jsonl are out of sync with the current schema or the live environment uses different agent configurations than the test harness. Update the JSONL file to reflect the expected output format and verify that the Cursor IDE is loading the same openai.yaml or gemini.toml files used during evaluation.

Which file controls the JSON structure the skill must return?

The skills/i-have-adhd/SKILL.md file defines both the prompt template and the expected outputs schema. If the LLM returns free-form text instead of JSON, inspect this file for missing code fences, incorrect field definitions, or schema drift between the prompt instructions and the formal output specification.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}