How Is Correctness Weighted in the i-have-adhd Evaluation Rubric?

In the ayghri/i-have-adhd repository, the Correctness dimension carries a weight of 35% in the evaluation rubric, making it the single most heavily weighted criterion.

The i-have-adhd repository implements a structured evaluation system for assessing model outputs across multiple quality dimensions. Understanding how Correctness is weighted helps interpret evaluation scores and prioritize improvements in model behavior.

The 35% Weight for Correctness in the Rubric

The evaluation rubric explicitly defines dimension weights in evals/rubric.md. Correctness receives the highest allocation at 35%, reflecting its priority as the foundational quality of any model response.

According to the rubric, Correctness measures factual and technical accuracy plus whether required details are preserved in the output. This emphasis ensures that even well-formatted, concise responses cannot score highly if they contain factual errors or omit critical information.

How Correctness Weight Is Implemented in the Scoring Script

The weight is programmatically enforced in scripts/run_evals.py through the WEIGHTS dictionary:

WEIGHTS = {
    "correctness": 0.35,
    "autonomy": 0.25,
    "actionability": 0.20,
    "safety": 0.10,
    "concision": 0.10,
}

This dictionary maps each dimension name to its decimal weight. When run_evals.py computes final scores, it multiplies each dimension's rating (scored 1–5) by its corresponding weight and sums the results.

Verifying the Correctness Weight Programmatically

You can extract and verify the Correctness weight directly from the source:

from pathlib import Path

# Path to the evaluation script relative to repository root

script_path = Path("scripts/run_evals.py")

# Parse the WEIGHTS dictionary (simplified extraction)

weights = {}
for line in script_path.read_text().splitlines():
    if line.strip().startswith('"correctness":'):
        weight = float(line.split(":")[1].strip().strip(","))
        weights["correctness"] = weight
        break

print(f"Correctness weight = {weights['correctness'] * 100}%")

# → Correctness weight = 35.0%

This confirms the 35% allocation matches the rubric documentation.

Complete Weight Distribution Across All Dimensions

The full weighting scheme prioritizes accuracy and self-sufficiency:

Dimension	Weight	Purpose
Correctness	35%	Factual accuracy and detail preservation
Autonomy	25%	Self-directed problem-solving without unnecessary clarification
Actionability	20%	Practical, implementable guidance
Safety	10%	Harm avoidance and appropriate boundaries
Concision	10%	Brevity without sacrificing completeness

The 10-percentage-point gap between Correctness (35%) and Autonomy (25%) demonstrates that technical accuracy takes precedence over independence in this evaluation framework. Even a perfectly autonomous response cannot compensate for correctness failures.

Impact on Aggregate Scoring

The weighting algorithm in run_evals.py applies these proportions uniformly across evaluation conditions (baseline vs. candidate comparisons). A Correctness score of 5 contributes 1.75 points to the weighted total (5 × 0.35), while a Concision score of 5 contributes only 0.50 points (5 × 0.10).

This mathematical structure means Correctness improvements yield 3.5× more score impact than equivalent improvements in Safety or Concision.

Summary

• Correctness is weighted at 35% in the i-have-adhd evaluation rubric, defined in evals/rubric.md.
• The same 35% weight appears in scripts/run_evals.py as "correctness": 0.35 in the WEIGHTS dictionary.
• This makes Correctness the highest-weighted dimension, 10 points above Autonomy (25%) and 25 points above Safety/Concision (10% each).
• Final scores multiply raw ratings (1–5) by these weights, so Correctness dominates aggregate results.

Frequently Asked Questions

What does the Correctness dimension actually measure in i-have-adhd?

Correctness evaluates factual and technical accuracy plus whether the output preserves all required details from the input or context. A response can be well-structured and safe yet score poorly on Correctness if it contains technical errors or omits critical information.

Why is Correctness weighted higher than Autonomy?

The 35% vs. 25% allocation reflects a prioritization hierarchy where accurate information outweighs self-directed behavior. According to the rubric structure in evals/rubric.md, a model that asks clarifying questions but eventually provides correct answers scores better than one that proceeds independently but produces incorrect outputs.

Can I modify the Correctness weight in my own evaluation runs?

Yes. Edit the WEIGHTS dictionary in scripts/run_evals.py to adjust the 0.35 value, or modify the table in evals/rubric.md to document your changes. The scoring script reads weights at runtime, so no recompilation is required.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}