i-Have-ADHD Skill Performance Metrics: Complete Evaluation Framework Explained

The i-have-adhd skill uses five weighted metrics—correctness (35%), autonomy (25%), actionability (20%), safety (10%), and concision (10%)—to calculate an overall weighted score for performance evaluation.

The ayghri/i-have-adhd repository implements a structured evaluation framework for measuring how effectively the skill assists users with ADHD. This framework, defined in scripts/run_evals.py, combines domain-specific quality criteria with automated scoring to produce reproducible performance reports and release gate decisions.

The Five Core Performance Metrics

The evaluation system captures five distinct dimensions of response quality. Each metric is scored on a 1–5 scale and contributes to a final weighted score.

Correctness (Weight: 0.35)

The correctness metric measures factual accuracy and relevance to the user's query. At 35% of the total weight, this is the dominant factor in performance assessment. Evaluators assign scores based on whether the response correctly addresses ADHD-related challenges with accurate information.

Autonomy (Weight: 0.25)

Autonomy evaluates how much the response reduces cognitive effort for the user—critical for an ADHD-focused tool. Higher scores indicate the skill handled complexity internally rather than burdening the user with decisions or follow-up requirements.

Actionability (Weight: 0.20)

The actionability metric scores the concreteness of next steps provided. Responses that include specific, timestamped, or immediately implementable instructions score higher than vague suggestions.

Safety (Weight: 0.10)

Safety checks for harmful advice regarding medication, unverified treatments, or other risky recommendations. While weighted lower, this metric serves as a hard blocker if critical failures occur.

Concision (Weight: 0.10)

Concision rewards brevity that preserves necessary detail. For ADHD users, this balances completeness against the cognitive cost of lengthy responses.

How the Weighted Score Is Calculated

The metric definitions and their numeric weights are declared in the WEIGHTS constant in scripts/run_evals.py (lines 20–25):

# From scripts/run_evals.py

WEIGHTS = {
    "correctness": 0.35,
    "autonomy": 0.25,
    "actionability": 0.20,
    "safety": 0.10,
    "concision": 0.10,
}

After validation via _validate_score (lines 90–92), which enforces the 1–5 range, the summarize_scores function (lines 64–68) computes averages and applies the weighted formula:

weighted_score = Σ(metric_average × weight)

The result is combined with a release gate pass/fail status in the final report structure.

Running the Evaluation

Command-Line Execution

python scripts/run_evals.py --cases evals/cases.jsonl --output results.json

Programmatic Score Analysis

from scripts.run_evals import summarize_scores

# `scores` is a list of dicts loaded from a JSON-Lines file

final_report = summarize_scores(scores)

print(final_report["conditions"]["candidate"]["weighted_score"])
print(final_report["release_gate"]["passed"])

Evaluation Data Format

Each evaluation row follows a structured schema. Here is a valid example:

{
  "case_id": "example-1",
  "trial": 1,
  "condition": "candidate",
  "correctness": 4,
  "autonomy": 5,
  "actionability": 3,
  "safety": 5,
  "concision": 4,
  "blocker": false,
  "notes": "All criteria met."
}

The condition field distinguishes between baseline and candidate versions for A/B comparisons. The blocker boolean can override the automated release gate if manual review identifies critical issues.

Key Source Files

• scripts/run_evals.py – Core evaluation logic, metric definitions (WEIGHTS), validation (_validate_score), and reporting (summarize_scores)

• skills/i-have-adhd/SKILL.md – Behavioral specification providing context for metric design decisions

Summary

• Five weighted metrics define i-have-adhd skill performance: correctness, autonomy, actionability, safety, and concision

• Correctness dominates at 35% weight, reflecting prioritization of accurate ADHD-related guidance

• Score range is fixed at 1–5 per metric, enforced by _validate_score in scripts/run_evals.py

• Weighted aggregation produces the final score used for release gate decisions

• Structured JSON-Lines format enables batch evaluation and automated comparison between skill versions

Frequently Asked Questions

What is the most important metric for the i-have-adhd skill?

Correctness carries the highest weight at 35%. This reflects the priority of accurate information when assisting users with ADHD, where misinformation about symptoms, treatments, or coping strategies could be particularly harmful.

How is the final score calculated from individual metrics?

The summarize_scores function in scripts/run_evals.py first computes the average of each metric across all evaluation rows, then applies the formula: weighted_score = Σ(metric_average × weight). This aggregated value determines whether the release gate passes.

Can an evaluation fail even if the weighted score is high?

Yes. The blocker field in evaluation rows allows manual override of the automated release gate. Additionally, the _validate_score function rejects any scores outside the 1–5 range, preventing calculation with invalid data.

Where are the metric weights defined in the source code?

The WEIGHTS dictionary is defined at lines 20–25 of scripts/run_evals.py. Modifying this constant changes how individual metrics contribute to the final weighted score for all subsequent evaluations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}