What Are Verification Loops and pass@k Metrics in Claude Code?

Verification loops are automated six-phase quality gates triggered by the /verify command that enforce build integrity and security checks, while pass@k metrics are statistical success rates calculated by the Eval Harness to quantify workflow reliability across multiple attempts.

The affaan-m/everything-claude-code repository defines these complementary mechanisms in skills/verification-loop/SKILL.md and skills/eval-harness/SKILL.md to transform AI-assisted coding from probabilistic generation into deterministic, measurable engineering. Together, they replace subjective "vibe checks" with automated validation and data-driven success thresholds.

How Verification Loops Work in Claude Code

Verification loops function as a mandatory quality checkpoint within the Claude Code workflow. Defined in skills/verification-loop/SKILL.md and invoked through the /verify slash command implemented in .opencode/commands/verify.md, these loops execute a hardened sequence of validation phases that must all pass before code is considered production-ready.

The Six Phases of a Verification Loop

Each verification cycle runs the following phases in strict order, halting immediately upon any failure:

1. Build verification – Confirms the project compiles cleanly using npm run build, pnpm build, or equivalent build scripts.

2. Type checking – Executes npx tsc --noEmit for TypeScript projects or pyright for Python to surface static type errors without emitting files.

3. Linting – Runs style enforcement via npm run lint (JavaScript/TypeScript) or ruff check (Python) to catch style violations and anti-patterns.

4. Test suite – Executes the full test suite with coverage reporting (npm run test -- --coverage), enforcing a mandatory minimum of 80% coverage.

5. Security scan – Greps the codebase for potential secrets (patterns like sk- or api_key) and stray console.log statements that could leak sensitive data.

6. Diff review – Displays git diff --stat to highlight the scope of changes and catch unintended modifications before commit.

If any phase fails, the loop terminates and emits a markdown report detailing the specific failure, requiring developer resolution before proceeding.

Verification Loop Output Format

The /verify command produces a concise markdown summary that serves as a "ready-for-PR" signal. A successful run appears as follows:

/verify

VERIFICATION REPORT
===================

Build:     PASS
Types:     PASS (0 errors)
Lint:      PASS (0 warnings)
Tests:     PASS (120/120 passed, 92% coverage)
Security:  PASS (0 issues)
Diff:      3 files changed

Overall:   READY for PR

Issues to Fix:
1. None – all checks passed.

Failed phases display actionable error messages extracted from the underlying tool output, allowing immediate remediation without manual log parsing.

Understanding pass@k Metrics in Claude Code

According to skills/eval-harness/SKILL.md, pass@k metrics provide a statistical framework for measuring how reliably a Claude Code workflow meets its success criteria across multiple independent attempts. These metrics were introduced in version 1.8.0 (documented in docs/releases/1.8.0/x-quote-eval-skills.md) to bring quantitative reliability engineering to AI-assisted development.

pass@1, pass@3, and pass^k Definitions

The Eval Harness defines three specific metric variants:

• pass@k – The probability that at least one attempt succeeds out of k total tries. Calculated as 1 minus the probability of k consecutive failures.

• pass@1 – The first-attempt success rate, indicating immediate task completion without retries. This measures single-shot capability.

• pass@3 – The probability of success within three attempts, accounting for stochastic variation in AI generation. The standard target for capability evaluations is pass@3 ≥ 0.90 (90% reliability).

• pass^3 (pass-cubed) – Requires all three attempts to succeed independently. This stricter metric demands pass^3 = 1.00 (100% consistency) for regression-critical paths where partial failure is unacceptable.

Defining Evaluations with pass@k Targets

Teams declare expected reliability thresholds within eval definitions using the following structure:

## EVAL DEFINITION: add‑auth

### Capability Evals

- [ ] User can register with email/password
- [ ] User can log in with valid credentials
- [ ] Invalid credentials return a 401 error

### Success Metrics

- pass@3 > 0.90
- pass^3 = 1.00   # all three runs must succeed for regression safety

Running /eval check add-auth executes the evaluation and computes the actual pass@k values against these thresholds, outputting a statistical report on workflow reliability.

CI/CD Integration and Automated Enforcement

Both verification loops and pass@k metrics integrate into CI/CD pipelines to gate deployments on objective quality criteria.

Gating Deployments with pass@k Checks

CI scripts can parse eval reports to enforce minimum reliability standards automatically:

# In a CI script after running the eval

if [[ $(cat eval-report.txt | grep "pass@3" | awk '{print $2}') < 0.90 ]]; then
  echo "FAIL: pass@3 below threshold"
  exit 1
fi

This pattern ensures that only workflows demonstrating ≥90% three-attempt reliability proceed to production, preventing flaky automation from reaching end users.

Continuous Verification Mode

The Verification Loop Skill supports continuous mode for active development sessions, automatically re-executing the six-phase validation workflow upon file changes. This provides real-time quality feedback without repetitive manual /verify invocations, as configured in skills/verification-loop/SKILL.md.

Summary

• Verification loops enforce deterministic quality gates through six automated phases (build, type-check, lint, test, security, diff) triggered by the /verify command in .opencode/commands/verify.md, with phase definitions stored in skills/verification-loop/SKILL.md.

• pass@k metrics quantify stochastic success rates, targeting pass@3 ≥ 0.90 for capability evaluations and pass^3 = 1.00 for regression-critical paths, as implemented in skills/eval-harness/SKILL.md.

• Both mechanisms originated in the affaan-m/everything-claude-code repository's v1.8.0 release and combine to provide deterministic static analysis (verification loops) with probabilistic reliability measurement (pass@k metrics).

• Integration patterns include CI/CD gates that parse eval reports and continuous verification modes that provide real-time development feedback.

Frequently Asked Questions

What triggers a verification loop in Claude Code?

Verification loops are triggered manually via the /verify slash command defined in .opencode/commands/verify.md, or automatically in continuous mode when file changes are detected. The command executes the six-phase validation sequence defined in skills/verification-loop/SKILL.md, halting immediately if any phase fails and displaying a markdown report with actionable fixes.

How is pass@3 calculated in Claude Code evaluations?

pass@3 is calculated by the Eval Harness as the probability that at least one of three independent attempts succeeds when executing a defined evaluation task. According to skills/eval-harness/SKILL.md, running /eval check [eval-name] generates this metric by attempting the task three times and reporting the success rate. Teams should target pass@3 values greater than 0.90 for reliable workflow validation.

What is the difference between pass@3 and pass^3 metrics?

pass@3 measures the probability of at least one success within three attempts (where fail-pass-pass counts as a successful trial), while pass^3 requires every single attempt to succeed (where fail-pass-pass counts as failure). As documented in skills/eval-harness/SKILL.md, use pass@3 > 0.90 for general capability evaluations and pass^3 = 1.00 for regression-critical paths requiring absolute consistency.

Can verification loops run automatically after every code change?

Yes. The Verification Loop Skill supports continuous mode operation that monitors the filesystem and automatically re-runs the full six-phase verification workflow upon detecting modifications. This configuration, detailed in skills/verification-loop/SKILL.md, provides immediate quality feedback during active development without requiring manual /verify invocation between edits.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}