How to Run Routing Regression Tests with test-routing.ps1 After Modifying routing.json

Run skills/scripts/test-routing.ps1 with the -Quick switch for fast iteration, or without it for full validation, after editing skills/config/routing.json to ensure all task-to-skill mappings remain correct.

The reverse-skill repository depends on routing.json as its single source of truth for routing task hints to appropriate skills. Any change to keyword patterns, priority ordering, or route definitions risks breaking existing mappings. The PowerShell script test-routing.ps1 provides automated regression testing to catch these issues before deployment.

How test-routing.ps1 Validates Routing Changes

The regression test script follows a four-stage pipeline defined in skills/scripts/test-routing.ps1:

1. Load benchmark — Reads skills/tests/routing-benchmark.json (default path at lines 21-24) containing hint-to-expected-route pairs.

2. Filter mode — Applies the -Quick switch to run only one case per route (lines 40-41), reducing execution time during iterative development.

3. Execute router — Invokes master-route.ps1 for each hint, extracts the primary route ID from generated route-scope.md (lines 55-58), and compares against expected values.

4. Report results — Aggregates outcomes into SUMMARY.txt and exits with code 0 for success or 1 for failure (lines 86-94). Detailed failures write to failures.txt (lines 90-92).

The underlying routing logic in master-route.ps1 (lines 22-30) reads routing.json, evaluates keyword matches, scores routes, and selects the primary route based on priority weights defined in the JSON configuration (lines 108-110 of routing.json).

Running the Regression Test: Complete Workflow

Follow this sequence after modifying routing.json:

Step	Action	Command
1	Edit configuration	Modify skills/config/routing.json with your routing changes
2	Quick validation	powershell -File skills/scripts/test-routing.ps1 -Quick
3	Full validation	powershell -NoProfile -ExecutionPolicy Bypass -File skills/scripts/test-routing.ps1
4	Review results	Open SUMMARY.txt in the generated log directory under %TEMP%
5	Fix failures	Adjust routing.json and repeat until exit code is 0

PowerShell Commands for Routing Regression Testing

Standard Full Test

Execute all ~162 benchmark cases to validate complete routing correctness:

powershell -NoProfile -ExecutionPolicy Bypass -File skills/scripts/test-routing.ps1

Quick Regression Test

Validate one case per route for rapid iteration during development:

powershell -File skills/scripts/test-routing.ps1 -Quick

Custom Benchmark and Logging

Point to alternative test data and specify output location:

powershell -File skills/scripts/test-routing.ps1 `
    -Benchmark .\custom-benchmark.json `
    -LogDir C:\Users\Me\reverse-skill-logs

Interpreting Test Output

A successful run produces output similar to:

=== test-routing | 162 cases (quick=False) ===
PASS R1 -> decompile APK with jadx apktool smali
PASS R2 -> analyze PE header structure
...
=== ROUTING TEST SUMMARY ===
TOTAL=162
PASS=162
FAIL=0
QUICK=False
LogDir=C:\Users\Me\AppData\Local\Temp\rs-routing-test-20260809-142530
OVERALL: PASS

Failed runs show mismatches explicitly:

[FAIL] hint='playwright browser automation' expect=R19 got=R3

Check failures.txt in the log directory for complete details on each mismatch.

Key Source Files in the Routing System

File	Purpose	Location
test-routing.ps1	Regression test orchestrator	skills/scripts/test-routing.ps1
master-route.ps1	Core routing engine	skills/scripts/master-route.ps1
routing.json	Route definitions and keyword mappings	skills/config/routing.json
routing-benchmark.json	Test dataset of hint-to-route pairs	skills/tests/routing-benchmark.json

Summary

• Always run test-routing.ps1 after editing routing.json to prevent regressions
• Use -Quick for fast feedback during iterative routing adjustments
• Check exit codes: 0 indicates success, 1 signals failures requiring fixes
• Inspect failures.txt in the generated log directory to identify specific hint-to-route mismatches
• Maintain benchmark parity: Update routing-benchmark.json when adding new routes or changing expected behaviors

Frequently Asked Questions

What does the -Quick parameter do in test-routing.ps1?

The -Quick switch filters the benchmark to run only one test case per defined route. According to lines 40-41 of test-routing.ps1, this reduces the full ~162-case suite to approximately 20-30 cases, enabling rapid validation during routing configuration iterations.

Why does test-routing.ps1 exit with code 1?

Exit code 1 indicates at least one routing test failed—meaning the actual route selected by master-route.ps1 differed from the expected route in routing-benchmark.json. Examine failures.txt in the generated log directory for specific mismatch details, adjust routing.json accordingly, and re-run.

Can I use a custom benchmark file instead of the default?

Yes. Pass the -Benchmark parameter with a path to your alternative JSON file. The benchmark must follow the same schema as routing-benchmark.json: an array of objects with hint and expect properties. Example: powershell -File skills/scripts/test-routing.ps1 -Benchmark .\my-tests.json.

Where are test results saved?

Results write to a timestamped directory under %TEMP% by default (pattern: rs-routing-test-YYYYMMDD-HHMMSS). The directory contains SUMMARY.txt with aggregate statistics and failures.txt with detailed error information. Use -LogDir to specify an alternative location.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}