How the 173-Case Routing Regression Benchmark Ensures Routing Accuracy in reverse-skill

The 173-case routing regression benchmark guarantees routing accuracy by requiring 100% test pass rate before any merge, with each case validating that a natural-language hint resolves to an exact expected skill identifier.

The reverse-skill repository maintains rigorous routing accuracy through a comprehensive regression test suite. This benchmark, stored as a JSON file with 173 test cases, serves as the definitive validation mechanism for the routing engine that maps user hints to skill modules.

The Core Benchmark Structure

The routing regression benchmark lives in skills/tests/routing-benchmark.json. Each entry represents a test case containing three fields:

• hint — the natural-language input that triggers routing
• expect — the skill identifier that must be returned
• quick — a boolean flag for test categorization

For the 73rd case (I73), the benchmark entry appears as:

{
  "hint": "some hint text …",
  "expect": "R7",
  "quick": false
}

This structure ensures every routing decision has a documented, verifiable expected outcome.

The Four-Stage Validation Process

Stage 1: Case Loading

The test runner loads the benchmark and extracts individual cases by index. In skills/scripts/test-routing.ps1 and its Bash equivalent skills/scripts/test-routing.sh, the runner iterates through all 173 entries sequentially.

Stage 2: Router Execution

The routing engine receives the hint and consults skills/config/routing.json — the authoritative routing table that maps hint patterns to skill IDs. The engine applies its matching logic and returns the resolved identifier.

Stage 3: Result Validation

The runner performs an exact string comparison between the returned skill ID and the case's expect field. Any mismatch causes immediate benchmark failure, blocking the CI pipeline.

Stage 4: Cross-Validation via Coherence Checks

The skills/scripts/verify-routing-coherence.ps1 script enforces additional integrity constraints:

• Minimum case count: Benchmark must contain at least 100 cases
• Well-formed identifiers: All expect values follow valid format rules
• Ghost prevention: Every expect ID must exist in routing.json

These checks prevent orphaned test cases and ensure benchmark maintainability.

Running the Benchmark

Windows (PowerShell)

# From repository root

powershell -NoProfile -ExecutionPolicy Bypass -File skills/scripts/test-routing.ps1

Linux/macOS (Bash)

bash skills/scripts/test-routing.sh

Programmatic Verification (Python)

This snippet demonstrates the validation logic used by the official runners:

import json
import subprocess
import pathlib

benchmark_path = pathlib.Path('skills/tests/routing-benchmark.json')
benchmark = json.loads(benchmark_path.read_text(encoding='utf-8-sig'))

case = benchmark['cases'][72]          # 73rd case (0-based index)

hint = case['hint']
expected = case['expect']

# Invoke PowerShell router

result = subprocess.check_output([
    'powershell', '-NoProfile', '-Command',
    f'& {{ .\\skills\\scripts\\router.ps1 -Hint \"{hint}\" }}'
]).decode().strip()

assert result == expected, f'Routing mismatch: {result} ≠ {expected}'
print('I73 case passed')

Key Files in the Verification Pipeline

File	Purpose
skills/tests/routing-benchmark.json	173-case test suite including I73
skills/scripts/test-routing.ps1	PowerShell benchmark runner
skills/scripts/test-routing.sh	Bash benchmark runner
skills/scripts/verify-routing-coherence.ps1	Sanity and integrity validator
skills/config/routing.json	Authoritative routing table

Summary

• The 173-case routing regression benchmark serves as the single source of truth for routing correctness in reverse-skill
• Each case validates exact matching between natural-language hints and skill identifiers
• Four-stage validation (load, execute, compare, cross-check) eliminates routing errors
• 100% pass rate requirement prevents regressions from reaching production
• Coherence verification in verify-routing-coherence.ps1 maintains benchmark integrity against schema drift

Frequently Asked Questions

What makes the 173-case benchmark a "regression" test?

A regression test detects unintended changes to previously working behavior. The 173-case benchmark captures 173 verified correct routing decisions as permanent fixtures. Any code change that alters routing logic must preserve all 173 outcomes, ensuring that fixes don't break existing functionality.

How does the I73 case specifically test routing accuracy?

The I73 case (73rd entry, index 72) validates that a specific natural-language hint resolves to skill identifier R7. This case, like all others, documents a real user scenario where a particular hint pattern must map to a predetermined skill module. Failure indicates the routing engine no longer recognizes that hint pattern correctly.

What happens if the benchmark fails during CI?

The repository blocks merge. Per the verification pipeline design, test-routing.ps1 or test-routing.sh returns a non-zero exit code on any mismatch. CI systems interpret this as a failed job, preventing the pull request from merging until the routing logic or benchmark data is corrected.

Can the benchmark detect missing or obsolete skill mappings?

Yes. The verify-routing-coherence.ps1 script explicitly checks that every expect ID in the benchmark exists in routing.json. This "ghost prevention" ensures test cases always reference valid skills, catching deletions or renames that would otherwise orphan test expectations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}