How Switchyard's StageRouter Chooses Between Efficient and Capable Tiers Using Tool Signals

Switchyard's StageRouter selects between efficient and capable tiers by inspecting tool-level signals such as requires_capable, respecting a configurable picker mode (efficient_first or capable_first), and falling back to the capable tier when model confidence drops below a calibrated threshold.

The StageRouter is a core routing algorithm in the Switchyard inference framework that dynamically routes requests between cost-efficient and high-capability model tiers. According to the NVIDIA-NeMo/Switchyard source code, this mechanism relies on explicit tool signals, confidence-based fallbacks, and configurable picker strategies to balance latency and quality.

Picker Mode Configuration

In crates/switchyard-runner/src/algorithm.rs (lines 353-360), the StageRouter exposes a picker configuration field that determines baseline tier preference. The implementation supports two distinct modes:

• efficient_first (default): The router prefers the efficient tier and only escalates to the capable tier when tool signals or confidence thresholds demand it.
• capable_first (experimental): The router prefers the capable tier, falling back to the efficient tier only when appropriate signals are present.

This configuration is evaluated at the start of every routing decision in crates/switchyard-server/src/stats/algorithms/stage_router.rs.

Tool-Level Signals for Tier Selection

Individual tools embedded in a request payload can emit explicit signals that override the picker mode. As implemented in stage_router.rs (lines 120-138), these signals use boolean flags:

• requires_capable: true: Forces the router to select the capable tier regardless of the picker mode.
• requires_efficient: true: Forces the router to select the efficient tier.

When the router inspects the request tools and encounters requires_capable: true, it immediately treats the request as a capable-tier requirement, bypassing the default efficient-first logic.

Confidence-Based Fallback Mechanism

Beyond explicit signals, the StageRouter monitors real-time model performance through metrics defined in crates/libsy/src/algorithms/util/stage.rs. The algorithm tracks:

• switchyard_stage_router_confidence: Model-specific confidence scores.
• switchyard_stage_router_probability: Routing probability distributions.

If a request's confidence falls below the configured confidence_threshold, the router automatically promotes the request to the capable tier even without an explicit tool signal. This fallback protects against low-confidence predictions in the efficient tier, as implemented in lines 241-250 of stage_router.rs.

Routing Decision Flow

The complete decision logic in crates/switchyard-server/src/stats/algorithms/stage_router.rs (lines 260-280) follows this execution path:

1. Read picker mode: Default to efficient_first unless configured otherwise.
2. Inspect tool signals: Check for requires_capable: true in any tool payload.
3. Evaluate confidence: If confidence is below confidence_threshold, route to capable tier.
4. Apply default: If no signals or confidence issues, remain in the efficient tier.
5. Record metrics: Emit routing decisions, probabilities, and confidence scores for observability.

Practical Implementation Example

The following Python examples demonstrate how to invoke the StageRouter with explicit tool signals using the Switchyard Python bindings exposed in crates/switchyard-py/src/libsy_bindings.rs. These examples assume a running Switchyard server at localhost:8000.

from switchyard_py import SwitchyardClient, RouteConfig

# Initialize client

client = SwitchyardClient(base_url="http://localhost:8000")

# Request requiring capable tier via tool signal

payload = {
    "messages": [{"role": "user", "content": "Explain quantum entanglement"}],
    "tools": [
        {
            "type": "code_analysis",
            "requires_capable": True  # Forces capable tier selection

        }
    ]
}

# Route using StageRouter (defaults to efficient_first)

response = client.run(payload, route=RouteConfig(algorithm="stage_router"))
print(response["model"])  # Returns a model from the capable tier

To route without the capable signal (staying in the efficient tier):

# Remove the signal to stay in efficient tier

payload["tools"][0].pop("requires_capable")
response = client.run(payload, route=RouteConfig(algorithm="stage_router"))
print(response["model"])  # Returns a model from the efficient tier

Summary

• The StageRouter uses a configurable picker mode (efficient_first by default) to establish baseline tier preference defined in crates/switchyard-runner/src/algorithm.rs.
• Tool signals (requires_capable, requires_efficient) override the picker mode, allowing per-request tier specification as processed in stage_router.rs lines 120-138.
• Confidence thresholds provide automatic fallback to the capable tier when model confidence is insufficient, monitored via switchyard_stage_router_confidence metrics.
• The implementation resides primarily in crates/switchyard-server/src/stats/algorithms/stage_router.rs with shared utilities in crates/libsy/src/algorithms/util/stage.rs.

Frequently Asked Questions

What is the default picker mode in Switchyard's StageRouter?

The default picker mode is efficient_first, configured in crates/switchyard-runner/src/algorithm.rs (lines 353-360). In this mode, the router attempts to use the efficient tier for all requests unless tool signals or confidence thresholds indicate otherwise. This default prioritizes latency and cost efficiency.

How do tool signals override the picker configuration?

When a tool in the request payload includes requires_capable: true, the StageRouter immediately routes to the capable tier regardless of the picker setting. This logic is implemented in stage_router.rs lines 120-138, where the router inspects tool metadata before applying the default picker logic. The signal effectively short-circuits the standard decision flow.

What happens when model confidence is below the threshold?

If the switchyard_stage_router_confidence metric falls below the configured confidence_threshold, the router automatically promotes the request to the capable tier. This behavior is defined in stage_router.rs lines 241-250. It provides a safety net against low-quality predictions without requiring explicit client-side signals.

Where is the StageRouter algorithm implemented in the Switchyard codebase?

The core algorithm resides in crates/switchyard-server/src/stats/algorithms/stage_router.rs, with shared utilities in crates/libsy/src/algorithms/util/stage.rs and configuration schemas in crates/switchyard-runner/src/algorithm.rs. Python bindings are exposed through crates/switchyard-py/src/libsy_bindings.rs. These files collectively implement the tier selection mechanism.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}