How to Run Routing Regression Tests After Modifying routing.json

Execute skills/scripts/test-routing.ps1 to validate that all task-to-skill mappings in routing.json still route correctly against the benchmark dataset.

The zhaoxuya520/reverse-skill repository centralizes routing logic in skills/config/routing.json. Any modification to this file risks breaking existing task classifications, so automated regression testing is essential. The test-routing.ps1 script provides comprehensive verification by replaying benchmark cases through the router and flagging discrepancies.

What the Routing Regression Test Validates

The regression suite confirms three critical properties of your routing.json changes:

• Keyword rule accuracy — Every hint in the benchmark resolves to the expected primary route
• Priority ordering correctness — The priority list in each route definition is respected during scoring
• Zero regression introduction — Previously working cases continue to produce identical results

The benchmark itself lives in skills/tests/routing-benchmark.json, containing approximately 162 hint-to-route pairs that exercise the full surface area of the routing logic.

How test-routing.ps1 Works

Benchmark Loading and Filtering

At lines 21–24, the script initializes the default benchmark path:

# From skills/scripts/test-routing.ps1 L21-L24

$Benchmark = Join-Path $PSScriptRoot ".." "tests" "routing-benchmark.json" | Resolve-Path
$benchmark = Get-Content $Benchmark | ConvertFrom-Json

The -Quick switch (lines 40–41) collapses the benchmark to one representative case per route, enabling rapid iteration during development.

Router Invocation and Result Extraction

For each benchmark case, the script:

1. Calls master-route.ps1 with the hint
2. Parses the generated route-scope.md
3. Extracts the primary route identifier (lines 55–58)

# Core verification logic from skills/scripts/test-routing.ps1 L55-L58

$routeScope = Get-Content "$WorkDir\route-scope.md" -Raw
if ($routeScope -match 'primary:\s*(\S+)') {
    $actualPrimary = $matches[1]
}

The actual primary is compared against the expected value from the benchmark. Mismatches are logged with full context for debugging.

Exit Codes and Summary Generation

Lines 86–94 handle result aggregation and final status:

• Exit code 0 — All cases passed
• Exit code 1 — One or more cases failed
• SUMMARY.txt — High-level statistics (total, pass, fail)
• failures.txt — Detailed mismatch records (lines 90–92)

Running the Regression Tests

Full Regression Suite

Execute all 162 benchmark cases to ensure comprehensive coverage:

powershell -NoProfile -ExecutionPolicy Bypass -File skills/scripts/test-routing.ps1

Quick Regression for Iteration

Use the -Quick flag during active development to reduce feedback time:

powershell -File skills/scripts/test-routing.ps1 -Quick

Custom Benchmark or Log Locations

Point to alternative test datasets or controlled output directories:

powershell -File skills/scripts/test-routing.ps1 `
    -Benchmark .\my-custom-benchmark.json `
    -LogDir C:\Users\Me\reverse-skill-logs

Interpreting Test Output

A typical execution produces console output followed by detailed logs:

=== test-routing | 162 cases (quick=False) ===
PASS R1 -> decompile APK with jadx apktool smali
...
[FAIL] hint='playwright browser automation' expect=R19 got=R3
...
=== ROUTING TEST SUMMARY ===
TOTAL=162
PASS=158
FAIL=4
QUICK=False
LogDir=C:\Users\Me\AppData\Local\Temp\rs-routing-test-20260809-142530
OVERALL: FAIL (4)

When failures occur, inspect failures.txt in the reported log directory to identify which keyword patterns need adjustment in routing.json.

How master-route.ps1 Consumes routing.json

The regression test exercises skills/scripts/master-route.ps1, which implements the actual routing engine. This script:

• Loads routing.json at lines 22–30
• Evaluates keyword matches against each route's patterns array
• Scores candidates and selects the primary based on the priority list (corresponding to lines 108–110 in the JSON structure)

Your modifications to routing.json directly affect this scoring pipeline, making the regression test the definitive validation mechanism.

Summary

• Invoke test-routing.ps1 after any routing.json edit to prevent regressions
• Use -Quick for rapid feedback during iterative development
• Check failures.txt in the log directory to debug mismatches
• Expect exit code 0 only when all benchmark cases pass
• Coordinate changes between routing.json and routing-benchmark.json when adding new routes

Frequently Asked Questions

What exit code does test-routing.ps1 return on failure?

The script exits with code 1 when any benchmark case fails, and 0 when all pass. This enables integration with CI/CD pipelines that depend on explicit status codes.

Where are regression test logs stored by default?

Logs are written to a timestamped subdirectory under %TEMP% with the pattern rs-routing-test-<yyyyMMdd-HHmmss>. Use -LogDir to override this location.

Can I add new test cases to the benchmark?

Yes. Append objects to skills/tests/routing-benchmark.json with hint and expected_route properties. The script loads this file dynamically, so no code changes are required.

How does the quick mode select representative cases?

When -Quick is specified, the script filters to one case per unique expected route, ensuring each route is exercised at least once without running the full 162-case suite.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}