What Does Autonomy Mean in the i-have-adhd Evaluation? A Technical Breakdown

In the i-have-adhd evaluation suite, "Autonomy" is a rubric dimension that measures an AI assistant's ability to self-direct through multi-stage workflows without continuous user prompting.

The i-have-adhd repository by ayghri implements a specialized evaluation framework designed specifically for AI assistants working with ADHD-related productivity tasks. Understanding what Autonomy means in this context requires examining how the evaluation rubric defines independent operation versus dependency on explicit instructions.

How Autonomy Is Defined in the Evaluation Rubric

The core definition of Autonomy resides in evals/rubric.md, which serves as the authoritative scoring guide for all evaluation runs. According to this file, an autonomous AI assistant demonstrates four critical capabilities:

• Self-direction — deciding the next logical step without waiting for step-by-step user commands
• Context maintenance — preserving and correctly reusing information from earlier in the conversation
• Error recovery — handling interruptions or incomplete inputs and continuing toward the goal
• Tool invocation — selecting and executing appropriate internal scripts without explicit instruction

The rubric evaluates these behaviors during actual task execution, assigning scores based on observed performance rather than claimed capabilities.

The Autonomy Scoring Criteria in Practice

During an evaluation run orchestrated by scripts/run_evals.py, the framework observes specific behavioral markers to determine Autonomy scores.

High Autonomy Indicators

When an assistant demonstrates strong Autonomy, it will:

1. Start the appropriate evaluation runner from a high-level request alone
2. Select correct agent configurations (OpenAI, Gemini, etc.) based on context
3. Update internal state variables such as progress counters without prompting
4. Progress through multi-step tasks without requesting clarification at each junction

Low Autonomy Warning Signs

Conversely, low Autonomy scores result from:

• Frequent requests for clarification on already-provided information
• Repetition of steps already completed
• Failure to advance toward stated goals without explicit "do this next" commands
• Inability to recover from minor errors without user intervention

These patterns directly contradict the repository's purpose as a self-sufficient productivity tool for ADHD-related workflows.

Where Autonomy Is Implemented in the Codebase

Three primary files govern how Autonomy is measured and enabled:

File Path	Function
evals/rubric.md	Defines the Autonomy dimension and scoring criteria
scripts/run_evals.py	Implements the evaluation orchestration logic that observes autonomous behavior
skills/i-have-adhd/SKILL.md	Documents expected autonomous capabilities for the skill

The run_evals.py script specifically contains the automation layer that determines whether an assistant qualifies as autonomous. It handles model selection, rubric loading, and progress tracking without requiring manual intervention between test cases.

Configuration Files That Enable Autonomous Operation

The autonomous runner relies on agent configuration files to make independent tooling decisions:

# From scripts/run_evals.py - the autonomous execution path

# The runner infers parameters without explicit per-parameter prompting

run(
    cases_path="evals/cases.jsonl",          # auto-locates test cases

    rubric_path="evals/rubric.md",           # loads Autonomy definition

    output_dir="results",                    # auto-determines output location

    model="gpt-4o-mini",                     # selects from configuration

)

The model selection itself demonstrates Autonomy — the runner reads configuration files to determine which backend to use:

# skills/i-have-adhd/agents/openai.yaml

model: gpt-4o-mini
temperature: 0.2
max_tokens: 1024

# skills/i-have-adhd/agents/gemini.toml

model = "gemini-1.5-flash"
temperature = 0.2

The autonomous system decides between these configurations based on availability, cost constraints, or task requirements without user micro-management.

Executing the Autonomous Evaluation

The complete autonomous workflow executes through a single entry point:

# No additional commands required after initial trigger

# The script internally handles: case loading → rubric parsing → 

# model selection → execution → result aggregation

$ python -m scripts.run_evals

This single-command execution exemplifies the Autonomy principle in action — the user provides intent ("run evaluation"), and the system handles all implementation details.

How Autonomy Differs from Other Rubric Dimensions

The i-have-adhd evaluation measures multiple quality dimensions, but Autonomy occupies a unique position. While Accuracy measures correct outputs and Helpfulness measures user satisfaction, Autonomy specifically addresses operational independence.

This distinction matters because ADHD-focused productivity tools must reduce cognitive load. An assistant requiring constant direction creates friction rather than alleviating it, regardless of output quality. The Autonomy dimension therefore serves as a gatekeeper criterion — insufficient autonomy disqualifies an assistant regardless of other strengths.

Summary

• Autonomy in i-have-adhd means independent operation through multi-stage workflows without continuous prompting
• The dimension is formally defined in evals/rubric.md and implemented through scripts/run_evals.py
• High-scoring assistants self-direct, maintain context, recover from errors, and invoke appropriate tools
• Configuration files in skills/i-have-adhd/agents/ enable automatic model selection
• This capability is essential for ADHD-focused tools that must minimize user cognitive overhead

Frequently Asked Questions

How is the Autonomy score calculated during evaluation?

The Autonomy score derives from behavioral observation during test case execution. The run_evals.py script tracks whether the assistant progresses through workflow stages independently, maintains state across steps, and recovers from simulated errors without explicit guidance. Each observed autonomous action increments the score; each dependency-seeking behavior decrements it.

What file contains the complete Autonomy definition?

The authoritative definition resides in evals/rubric.md. This markdown file specifies the precise criteria separating high, medium, and low Autonomy ratings, including specific behavioral examples for each scoring tier. All evaluation runs reference this file through the rubric_path parameter.

Can an assistant with low Autonomy still pass the i-have-adhd evaluation?

No. The repository's design philosophy treats Autonomy as fundamental to its mission. An assistant scoring low on this dimension would fail regardless of Accuracy or Helpfulness scores, because the tool specifically targets users who need reduced cognitive overhead from constant direction-giving.

How does the autonomous runner select between OpenAI and Gemini configurations?

The scripts/run_evals.py logic examines availability, cost parameters, and task requirements specified in the evaluation case, then automatically selects from configurations in skills/i-have-adhd/agents/openai.yaml or skills/i-have-adhd/agents/gemini.toml without requiring explicit user model selection per run.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}