How Regression Testing in reverse-skill Validates Routing Logic Automatically

Regression testing in reverse-skill is performed through an automated benchmark suite defined in skills/tests/routing-benchmark.json and executed by cross-platform PowerShell and Bash harnesses that verify routing output against expected skill identifiers.

The zhaoxuya520/reverse-skill repository maintains routing integrity through deterministic regression testing. Every modification to skills/config/routing.json or individual skill scripts must satisfy the entire benchmark catalogue to prevent regressions. The framework exercises both Windows and Unix-like platforms through dedicated test harnesses that isolate execution environments and validate output formats.

The Regression Test Architecture

Benchmark Definition in routing-benchmark.json

The canonical test catalogue lives at skills/tests/routing-benchmark.json. This JSON file enumerates every routing scenario as an object containing three fields:

• hint – The user-level query string passed to the router
• expect – The primary skill identifier expected in the output
• quick – A boolean flag marking the case as part of the minimal smoke-test subset

When developers append new routing logic, they extend the cases array in this file. The next automated run will automatically validate the new scenario without requiring changes to the test harness logic.

Cross-Platform Test Harnesses

Two scripts provide identical regression coverage across operating systems:

1. skills/scripts/test-routing.ps1 – PowerShell implementation for Windows environments
2. skills/scripts/test-routing.sh – Bash implementation for Linux and macOS

Both harnesses perform the same core operations: they spawn isolated temporary directories, invoke the master router with benchmark hints, parse the generated route-scope.md, and compare extracted skill IDs against expected values.

How the Test Harnesses Execute Routing Scenarios

PowerShell Implementation

skills/scripts/test-routing.ps1 orchestrates the Windows regression pipeline through the following steps:

1. Loads the benchmark JSON and iterates over the cases array
2. Creates a temporary directory for each test case to ensure isolation
3. Invokes skills/scripts/master-route.ps1 via the host executable with the --hint parameter
4. Parses route-scope.md using skills/scripts/lib/RouteScope.ps1 to extract the Id field
5. Compares the extracted ID against the expect value from the benchmark
6. Accumulates PASS/FAIL counts and writes a summary log to the designated directory

The script supports a -Quick switch that filters execution to only cases where "quick": true, enabling rapid smoke testing during development.

# Run the full regression suite on Windows

powershell -NoProfile -ExecutionPolicy Bypass -File skills/scripts/test-routing.ps1

# Execute only the minimal smoke-test set

powershell -File skills/scripts/test-routing.ps1 -Quick

Bash Implementation

skills/scripts/test-routing.sh provides equivalent functionality for Unix platforms with additional path-resolution validation:

1. Determines the available Python interpreter (lines 10-15)
2. Embeds a Python snippet to parse the JSON benchmark
3. Executes skills/scripts/master-route.sh for each case with appropriate --hint arguments
4. Inspects route-scope.md for the - primary: skill identifier
5. Performs exit-code validation and aggregates results

The Bash version includes a specific regression check absent from the PowerShell implementation: it verifies that the router writes output to the caller’s project directory when --out-dir is omitted, ensuring the tool does not pollute the installed package directory.

# Run the full regression suite on Linux/macOS

bash skills/scripts/test-routing.sh

Special Regression Checks

Default-Root Path Validation

Beyond per-case routing validation, the Bash test harness implements a default-root regression check (lines 70-78). This test ensures that when master-route.sh runs without an explicit --out-dir argument, the generated route-scope.md appears within the caller’s project workspace rather than the reverse-skill installation directory.

This guard prevents subtle path-resolution bugs where the router might incorrectly resolve output locations relative to its own source tree instead of the user’s working directory. The test explicitly verifies file existence in the temporary directory after execution completes.

Extending the Regression Suite

Adding new routing scenarios requires only modifying the benchmark JSON:

{
  "hint": "new reverse-engineering scenario",
  "expect": "R99",
  "quick": true
}

Append this object to the cases array in skills/tests/routing-benchmark.json. Both harnesses will automatically incorporate the new case in subsequent runs, validating that the routing logic correctly maps the new hint to skill R99.

Summary

• Regression testing in reverse-skill relies on skills/tests/routing-benchmark.json as the single source of truth for routing scenarios.
• Cross-platform harnesses at skills/scripts/test-routing.ps1 and skills/scripts/test-routing.sh execute the benchmark deterministically on Windows and Unix.
• Each test run creates isolated temporary directories, invokes the master router, and parses route-scope.md to verify the primary skill identifier.
• The Bash harness includes a default-root regression check that validates output directory resolution when --out-dir is omitted.
• The -Quick flag filters execution to smoke-test cases marked with "quick": true for rapid validation cycles.

Frequently Asked Questions

Where are the regression test cases defined in reverse-skill?

All test cases are defined in skills/tests/routing-benchmark.json. Each entry specifies a user hint, an expected skill ID, and an optional quick-test flag. The JSON structure allows the PowerShell and Bash harnesses to load and iterate over scenarios without hardcoding test logic.

How do I run only the quick smoke tests?

Pass the -Quick switch to the PowerShell harness: powershell -File skills/scripts/test-routing.ps1 -Quick. This filters the benchmark to only cases where the quick property is true, significantly reducing execution time for rapid validation during development.

What is the default-root regression check?

The default-root regression check is a specific validation in skills/scripts/test-routing.sh (lines 70-78) that ensures the router writes output to the caller’s project directory when no --out-dir argument is provided. This prevents the router from incorrectly placing files in its own installation directory.

Which files parse the routing output during testing?

The PowerShell harness uses skills/scripts/lib/RouteScope.ps1 to extract the Id field from route-scope.md, while the Bash harness uses grep-based parsing to locate the - primary: skill identifier. Both methods verify that the router selected the expected skill defined in the benchmark.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}