# What Are Verification Loops and pass@k Metrics in Claude Code?

> Understand verification loops and pass@k metrics in Claude Code. Learn about automated quality gates and reliability statistics to improve your workflow.

- Repository: [Affaan Mustafa/everything-claude-code](https://github.com/affaan-m/everything-claude-code)
- Tags: deep-dive
- Published: 2026-03-20

---

**Verification loops are automated six-phase quality gates triggered by the `/verify` command that enforce build integrity and security checks, while pass@k metrics are statistical success rates calculated by the Eval Harness to quantify workflow reliability across multiple attempts.**

The `affaan-m/everything-claude-code` repository defines these complementary mechanisms in [`skills/verification-loop/SKILL.md`](https://github.com/affaan-m/everything-claude-code/blob/main/skills/verification-loop/SKILL.md) and [`skills/eval-harness/SKILL.md`](https://github.com/affaan-m/everything-claude-code/blob/main/skills/eval-harness/SKILL.md) to transform AI-assisted coding from probabilistic generation into deterministic, measurable engineering. Together, they replace subjective "vibe checks" with automated validation and data-driven success thresholds.

## How Verification Loops Work in Claude Code

Verification loops function as a mandatory quality checkpoint within the Claude Code workflow. Defined in [`skills/verification-loop/SKILL.md`](https://github.com/affaan-m/everything-claude-code/blob/main/skills/verification-loop/SKILL.md) and invoked through the `/verify` slash command implemented in [`.opencode/commands/verify.md`](https://github.com/affaan-m/everything-claude-code/blob/main/.opencode/commands/verify.md), these loops execute a hardened sequence of validation phases that must all pass before code is considered production-ready.

### The Six Phases of a Verification Loop

Each verification cycle runs the following phases in strict order, halting immediately upon any failure:

1. **Build verification** – Confirms the project compiles cleanly using `npm run build`, `pnpm build`, or equivalent build scripts.

2. **Type checking** – Executes `npx tsc --noEmit` for TypeScript projects or `pyright` for Python to surface static type errors without emitting files.

3. **Linting** – Runs style enforcement via `npm run lint` (JavaScript/TypeScript) or `ruff check` (Python) to catch style violations and anti-patterns.

4. **Test suite** – Executes the full test suite with coverage reporting (`npm run test -- --coverage`), enforcing a mandatory minimum of **80% coverage**.

5. **Security scan** – Greps the codebase for potential secrets (patterns like `sk-` or `api_key`) and stray `console.log` statements that could leak sensitive data.

6. **Diff review** – Displays `git diff --stat` to highlight the scope of changes and catch unintended modifications before commit.

If any phase fails, the loop terminates and emits a markdown report detailing the specific failure, requiring developer resolution before proceeding.

### Verification Loop Output Format

The `/verify` command produces a concise markdown summary that serves as a "ready-for-PR" signal. A successful run appears as follows:

```bash
/verify

```

```markdown
VERIFICATION REPORT
===================

Build:     PASS
Types:     PASS (0 errors)
Lint:      PASS (0 warnings)
Tests:     PASS (120/120 passed, 92% coverage)
Security:  PASS (0 issues)
Diff:      3 files changed

Overall:   READY for PR

Issues to Fix:
1. None – all checks passed.

```

Failed phases display actionable error messages extracted from the underlying tool output, allowing immediate remediation without manual log parsing.

## Understanding pass@k Metrics in Claude Code

According to [`skills/eval-harness/SKILL.md`](https://github.com/affaan-m/everything-claude-code/blob/main/skills/eval-harness/SKILL.md), pass@k metrics provide a statistical framework for measuring how reliably a Claude Code workflow meets its success criteria across multiple independent attempts. These metrics were introduced in version 1.8.0 (documented in [`docs/releases/1.8.0/x-quote-eval-skills.md`](https://github.com/affaan-m/everything-claude-code/blob/main/docs/releases/1.8.0/x-quote-eval-skills.md)) to bring quantitative reliability engineering to AI-assisted development.

### pass@1, pass@3, and pass^k Definitions

The Eval Harness defines three specific metric variants:

- **pass@k** – The probability that at least one attempt succeeds out of *k* total tries. Calculated as 1 minus the probability of *k* consecutive failures.

- **pass@1** – The first-attempt success rate, indicating immediate task completion without retries. This measures single-shot capability.

- **pass@3** – The probability of success within three attempts, accounting for stochastic variation in AI generation. The standard target for capability evaluations is **pass@3 ≥ 0.90** (90% reliability).

- **pass^3** (pass-cubed) – Requires **all three attempts** to succeed independently. This stricter metric demands **pass^3 = 1.00** (100% consistency) for regression-critical paths where partial failure is unacceptable.

### Defining Evaluations with pass@k Targets

Teams declare expected reliability thresholds within eval definitions using the following structure:

```markdown

## EVAL DEFINITION: add‑auth

### Capability Evals

- [ ] User can register with email/password
- [ ] User can log in with valid credentials
- [ ] Invalid credentials return a 401 error

### Success Metrics

- pass@3 > 0.90
- pass^3 = 1.00   # all three runs must succeed for regression safety

```

Running `/eval check add-auth` executes the evaluation and computes the actual pass@k values against these thresholds, outputting a statistical report on workflow reliability.

## CI/CD Integration and Automated Enforcement

Both verification loops and pass@k metrics integrate into CI/CD pipelines to gate deployments on objective quality criteria.

### Gating Deployments with pass@k Checks

CI scripts can parse eval reports to enforce minimum reliability standards automatically:

```bash

# In a CI script after running the eval

if [[ $(cat eval-report.txt | grep "pass@3" | awk '{print $2}') < 0.90 ]]; then
  echo "FAIL: pass@3 below threshold"
  exit 1
fi

```

This pattern ensures that only workflows demonstrating ≥90% three-attempt reliability proceed to production, preventing flaky automation from reaching end users.

### Continuous Verification Mode

The Verification Loop Skill supports continuous mode for active development sessions, automatically re-executing the six-phase validation workflow upon file changes. This provides real-time quality feedback without repetitive manual `/verify` invocations, as configured in [`skills/verification-loop/SKILL.md`](https://github.com/affaan-m/everything-claude-code/blob/main/skills/verification-loop/SKILL.md).

## Summary

- **Verification loops** enforce deterministic quality gates through six automated phases (build, type-check, lint, test, security, diff) triggered by the `/verify` command in [`.opencode/commands/verify.md`](https://github.com/affaan-m/everything-claude-code/blob/main/.opencode/commands/verify.md), with phase definitions stored in [`skills/verification-loop/SKILL.md`](https://github.com/affaan-m/everything-claude-code/blob/main/skills/verification-loop/SKILL.md).

- **pass@k metrics** quantify stochastic success rates, targeting `pass@3 ≥ 0.90` for capability evaluations and `pass^3 = 1.00` for regression-critical paths, as implemented in [`skills/eval-harness/SKILL.md`](https://github.com/affaan-m/everything-claude-code/blob/main/skills/eval-harness/SKILL.md).

- Both mechanisms originated in the `affaan-m/everything-claude-code` repository's v1.8.0 release and combine to provide deterministic static analysis (verification loops) with probabilistic reliability measurement (pass@k metrics).

- Integration patterns include CI/CD gates that parse eval reports and continuous verification modes that provide real-time development feedback.

## Frequently Asked Questions

### What triggers a verification loop in Claude Code?

Verification loops are triggered manually via the `/verify` slash command defined in [`.opencode/commands/verify.md`](https://github.com/affaan-m/everything-claude-code/blob/main/.opencode/commands/verify.md), or automatically in continuous mode when file changes are detected. The command executes the six-phase validation sequence defined in [`skills/verification-loop/SKILL.md`](https://github.com/affaan-m/everything-claude-code/blob/main/skills/verification-loop/SKILL.md), halting immediately if any phase fails and displaying a markdown report with actionable fixes.

### How is pass@3 calculated in Claude Code evaluations?

pass@3 is calculated by the Eval Harness as the probability that at least one of three independent attempts succeeds when executing a defined evaluation task. According to [`skills/eval-harness/SKILL.md`](https://github.com/affaan-m/everything-claude-code/blob/main/skills/eval-harness/SKILL.md), running `/eval check [eval-name]` generates this metric by attempting the task three times and reporting the success rate. Teams should target pass@3 values greater than 0.90 for reliable workflow validation.

### What is the difference between pass@3 and pass^3 metrics?

pass@3 measures the probability of at least one success within three attempts (where fail-pass-pass counts as a successful trial), while pass^3 requires every single attempt to succeed (where fail-pass-pass counts as failure). As documented in [`skills/eval-harness/SKILL.md`](https://github.com/affaan-m/everything-claude-code/blob/main/skills/eval-harness/SKILL.md), use pass@3 > 0.90 for general capability evaluations and pass^3 = 1.00 for regression-critical paths requiring absolute consistency.

### Can verification loops run automatically after every code change?

Yes. The Verification Loop Skill supports continuous mode operation that monitors the filesystem and automatically re-runs the full six-phase verification workflow upon detecting modifications. This configuration, detailed in [`skills/verification-loop/SKILL.md`](https://github.com/affaan-m/everything-claude-code/blob/main/skills/verification-loop/SKILL.md), provides immediate quality feedback during active development without requiring manual `/verify` invocation between edits.