How to Run the Routing Regression Tests for reverse-skill

Run skills/scripts/test-routing.ps1 with PowerShell to execute the full routing regression suite, or add -Quick to test only the minimal subset of cases.

The routing regression tests in the zhaoxuya520/reverse-skill repository ensure that every hint → PRIMARY mapping defined in the benchmark file still resolves correctly after any change to the routing rules. This guide covers the complete workflow from prerequisites through interpreting test results.

What the Routing Regression Tests Verify

The regression suite validates that master-route.ps1 consistently returns the expected PRIMARY skill identifier for every registered routing hint. Each test case in routing-benchmark.json contains:

• A hint — the user input pattern
• An expected PRIMARY — the skill route that should be selected
• A quick flag — whether this case is included in fast validation runs

When you modify skills/config/routing.json, these tests confirm you haven't broken existing mappings.

Prerequisites

Before running the tests, ensure your environment meets these requirements:

• PowerShell 5.1+ (Windows) or PowerShell Core (Linux/macOS)
• The repository's tool index must be refreshed so external dependencies are detected
• Access to %TEMP% on Windows or /tmp on Unix systems for log output

Step 1: Refresh the Tool Index

The reverse-skill repository maintains an indexed registry of available tools. Run this once after cloning:

# Windows

powershell -File skills/scripts/refresh-tool-index.ps1

# Linux / macOS

bash skills/scripts/refresh-tool-index.sh

This step is required before master-route.ps1 can correctly resolve routing decisions.

Step 2: Execute the Routing Regression Test

The primary test runner is skills/scripts/test-routing.ps1. It handles benchmark loading, temporary workspace creation, per-case execution, and result aggregation.

Run the Full Benchmark (Default, 163 Cases)

powershell -NoProfile -ExecutionPolicy Bypass -File skills/scripts/test-routing.ps1

This validates every hint-to-route mapping in the benchmark file. Execution time depends on case complexity.

Run Quick Mode (Minimal Subset)

powershell -File skills/scripts/test-routing.ps1 -Quick

Quick mode executes only cases marked "quick": true in routing-benchmark.json — approximately one case per route. Use this for rapid feedback during development.

Custom Benchmark and Log Directory

powershell -File skills/scripts/test-routing.ps1 `
    -Benchmark path\to\my-benchmark.json `
    -LogDir C:\temp\my-routing-log

The -Benchmark parameter overrides the default location. The -LogDir parameter controls where test artifacts are written (defaults to a timestamped subdirectory of %TEMP% or /tmp).

Understanding Test Output

A successful run produces output similar to:

=== test-routing | 163 cases (quick=False) ===
PASS R1 -> decompile APK with jadx apktool smali
...
=== ROUTING TEST SUMMARY ===
TOTAL=163
PASS=163
FAIL=0
QUICK=False
LogDir=C:\Users\<name>\AppData\Local\Temp\rs-routing-test-20260810-152300
OVERALL: ALL PASS (163)

Log Directory Contents

Each run creates a unique directory containing:

File	Purpose
cases.txt	Per-case execution details
SUMMARY.txt	Aggregated statistics
failures.txt	List of failed cases (only if FAIL > 0)

Interpreting Failures

If any case fails, failures.txt contains lines like:

FAIL hint='disassemble ARM binary' expect=R7 got=EXC:tool_not_found

The exit code is 1 on failure. Use $LASTEXITCODE in PowerShell to detect this programmatically:

powershell -NoProfile -ExecutionPolicy Bypass -File skills/scripts/test-routing.ps1
if ($LASTEXITCODE -eq 0) {
    Write-Host "All routing cases passed."
} else {
    Write-Host "Some routing cases failed – inspect $((Get-ChildItem $env:TEMP -Filter 'rs-routing-test-*' | Sort-Object LastWriteTime -Descending | Select-Object -First 1).FullName)"
}

Key Files in the Routing Test Architecture

File	Role
skills/scripts/test-routing.ps1	Test orchestrator — loads benchmark, invokes master-route.ps1, aggregates results (lines 21–96)
skills/tests/routing-benchmark.json	JSON array of test cases with hint, expected PRIMARY, and quick flag (lines 9–13+)
skills/scripts/master-route.ps1	Core routing logic that reads routing.json and produces route-scope.md
skills/config/routing.json	Canonical routing rules (R0–R40); any modification requires regression verification

Automated Testing in CI Pipelines

Integrate the routing regression tests into your build process:

# Run quick validation on every commit

powershell -File skills/scripts/test-routing.ps1 -Quick
if ($LASTEXITCODE -ne 0) { throw "Routing regression failed" }

# Run full suite before releases

powershell -NoProfile -ExecutionPolicy Bypass -File skills/scripts/test-routing.ps1
if ($LASTEXITCODE -ne 0) { throw "Full routing regression failed" }

The -NoProfile and -ExecutionPolicy Bypass flags ensure consistent execution in restricted environments.

Summary

• Primary test runner: skills/scripts/test-routing.ps1
• Benchmark source: skills/tests/routing-benchmark.json — the single source of truth for regression cases
• Quick mode: Use -Quick for rapid validation of one case per route
• Success indicator: Exit code 0 with PASS equal to TOTAL
• Failure diagnosis: Check failures.txt in the generated log directory for detailed mismatch information
• Prerequisite step: Refresh tool index with refresh-tool-index.ps1 or .sh before first run

Frequently Asked Questions

What PowerShell version is required for the routing regression tests?

PowerShell 5.1 or later on Windows, or PowerShell Core on Linux and macOS. The script uses modern parameter binding and temp file handling that requires these versions.

Where does the routing regression test get its expected results?

From skills/tests/routing-benchmark.json, a JSON file containing 163 test cases as of the current repository state. Each case specifies an input hint, the expected PRIMARY route identifier, and whether it belongs to the quick test subset.

Can I run the routing regression tests without executing the full 163 cases?

Yes. Add the -Quick switch to test-routing.ps1 to run only cases where "quick": true in the benchmark file — typically one representative case per routing rule rather than exhaustive coverage.

What should I do if a routing test fails after modifying routing.json?

Examine failures.txt in the generated log directory to identify which hint produced an unexpected PRIMARY. The file shows expected versus actual results, including exception details if routing failed entirely. Revert your change or update the benchmark if the new behavior is intentional.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}