How to Debug Routing Misclassifications Using the Verification Scripts in Reverse‑Skill

Run the test suite to generate a log, then execute verify-routing-coherence.ps1 to identify mismatches between detected and expected routes in skills/config/routing.json.

The reverse‑skill project routes user requests to security skills using three orthogonal dimensions: target type, user intent, and tool chain. When requests are mis‑routed, the repository provides dedicated verification scripts to compare actual routing results against expected definitions. This guide walks through the debugging workflow using the actual source files in zhaoxuya520/reverse‑skill.

Run the Routing Test Suite

Start by executing the platform‑appropriate test harness. This script drives the router with benchmark hints and records outcomes.

Linux/macOS:

bash skills/scripts/test-routing.sh

Windows:

powershell -NoProfile -ExecutionPolicy Bypass -File skills/scripts/test-routing.ps1

Both scripts invoke the master route entry point (master-route.sh or master-route.ps1) using hints defined in skills/config/routing-benchmark.json. For each hint, the script logs:

• Detected route: The skill actually selected by the router
• Expected route: The value defined in skills/config/routing.json

The output is written to routing.log.csv for subsequent analysis.

Verify Routing Coherence

The verify-routing-coherence.ps1 script parses the CSV log and surfaces discrepancies. Run it after the test suite completes:

powershell -NoProfile -ExecutionPolicy Bypass -File skills/scripts/verify-routing-coherence.ps1

This PowerShell script performs three functions:

1. Compares detected vs. expected routes for every test hint
2. Prints a concise table showing the hint, mismatched route, and corresponding routing.json entry
3. Suggests nearest matches using Levenshtein distance to catch typos or outdated entries

Sample output:

Hint: "run privilege escalation on Windows"
Detected:  powershell/priv-escalate
Expected:  windows/priv-escalate
Suggestion:  windows/priv-escalate  (nearest match)

The Suggestion field helps determine whether the issue is a simple typo, a stale benchmark hint, or a deeper logic problem in the routing implementation.

Inspect and Fix the Routing Definition

Routing rules live in skills/config/routing.json. Each entry contains:

• hint: Pattern string matched against user requests
• target: Platform identifier (windows, linux, kali)
• intent: Action category (recon, exploit, priv-escalate)
• toolChain: Tool identifier (e.g., powershell, bash, metasploit)

When verify-routing-coherence.ps1 reports a mismatch, locate the offending entry:

{
  "hint": "run privilege escalation on Windows",
  "target": "windows",
  "intent": "priv-escalate",
  "toolChain": "powershell"
}

Adjust fields as needed, then re‑run the test suite and verification script until no mismatches remain.

Common Causes of Routing Misclassifications

Persistent mismatches typically stem from three sources:

• Stale entries: The benchmark hint in routing-benchmark.json was modified but routing.json was not updated
• Over‑broad regexes: A hint pattern matches multiple intents, causing the router to select the wrong skill
• Missing platform overrides: The route differs between Windows, Linux, and Kali; verify the target field matches the execution environment

Debugging Workflow Summary

Step	Script	Purpose
1	test-routing.sh / test-routing.ps1	Execute benchmark, generate routing.log.csv
2	verify-routing-coherence.ps1	Identify mismatches and suggest fixes
3	Edit skills/config/routing.json	Correct rules, then repeat steps 1‑2

Summary

• Primary verification script: skills/scripts/verify-routing-coherence.ps1 analyzes test output and flags routing mismatches
• Test harnesses: test-routing.sh (Linux/macOS) and test-routing.ps1 (Windows) generate the required CSV log
• Configuration source: skills/config/routing.json defines canonical routing rules across target, intent, and tool chain dimensions
• Debugging cycle: Run test → verify coherence → fix JSON → repeat until zero mismatches

Frequently Asked Questions

What file does verify-routing-coherence.ps1 read?

The script reads routing.log.csv, which is generated by test-routing.ps1 or test-routing.sh. This CSV contains detected routes, expected routes, and the original hint strings for each benchmark case.

Can I run the verification script on Linux or macOS?

verify-routing-coherence.ps1 is a PowerShell script, but PowerShell Core (pwsh) runs cross‑platform. Install PowerShell Core, then execute: pwsh skills/scripts/verify-routing-coherence.ps1

How does the script suggest nearest matches?

It calculates Levenshtein distance between the detected route string and all valid routes in routing.json, then returns the closest match. This helps identify typos and near‑miss configurations quickly.

Where are the benchmark hints defined?

Test cases reside in skills/config/routing-benchmark.json. Add new hints here to expand test coverage for custom routing scenarios.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}