How to Configure an Escalation Router with an LLM Judge for Automatic Model Tier Promotion in Switchyard

Switchyard's escalation router lets a low-cost "efficient" tier model automatically hand off requests to a higher-cost "capable" tier when an LLM-based escalation judge decides the request exceeds the efficient model's capabilities.

Switchyard's escalation router is a cost-optimization pattern that routes easy requests to cheap models and promotes hard requests to capable models based on runtime evaluation. By configuring an LLM judge as the promotion gatekeeper, you avoid wasting expensive model calls on trivial tasks while ensuring complex queries receive adequate compute.

Architecture of the Escalation Router

The escalation system comprises four coordinated components:

Component	Role	Source Location
Stage router (stage_router)	Orchestrates the two-tier routing decision between fast and strong sub-routers	switchyard_rust/libsy.py
Escalation judge	Lightweight LLM that renders escalate or stay verdicts on each request	crates/libsy/src/algorithms/util/escalation.rs
Escalation policy	Defines how many consecutive "escalate" signals trigger actual promotion	crates/libsy/src/algorithms/util/escalation.rs
Handoff notes	Optional contextual strings passed between tiers during promotion/demotion	crates/libsy/src/algorithms/util/stage.rs

The execution flow follows a strict sequence: the fast router processes the request first, the escalation judge evaluates whether promotion is warranted, and—if the policy threshold is met—the strong router completes the response. This architecture separates routing orchestration from evaluation logic, allowing independent iteration on judge prompts and promotion policies.

Step-by-Step Configuration

1. Define the Escalation Route in TOML

Create a deployment configuration file (e.g., production_routes.toml) that declares an escalation-mode route with both sub-routers specified:

[routes.production_escalation]
id = "my-org/production-escalation"
mode = "escalation"

fast = { type = "random", name = "gpt-4o-mini" }
strong = { type = "llm_task_classifier", model = "openrouter/openai/gpt-4o" }

The mode = "escalation" directive activates the stage router with promotion logic enabled.

2. Configure the LLM Judge Parameters

Within the same route block, add the escalation table with judge-specific settings:

[routes.production_escalation.escalation]
judge = "openrouter/openai/gpt-4o-mini"
confirmations = 2
max_output_tokens = 256
prompt = "Evaluate if this request requires advanced reasoning. Reply ESCALATE or STAY."

Critical parameters include:

• judge: Model identifier for the evaluation LLM—typically a cheap, fast model like gpt-4o-mini
• confirmations: Consecutive "escalate" verdicts required before actual promotion (default: 1)
• max_output_tokens: Hard limit on judge response length
• prompt: Optional override for the built-in escalation evaluation prompt

Higher confirmations values reduce false promotions caused by judge noise but increase latency for legitimately hard requests.

3. Add Handoff Notes for Context Preservation

Configure notes that inform each tier when promotions or demotions occur:

escalation_note = "🚀 PROMOTED: This request was escalated from the efficient tier due to complexity."
deescalation_note = "🔁 RETURNED: Request completed; resuming efficient tier for follow-ups."
only_on_wrong_signal_escalation = true

When only_on_wrong_signal_escalation = true (the default), notes attach only to signal-driven escalations rather than all routing decisions. This prevents note pollution during manual tier selection.

4. Launch Switchyard with the Custom Configuration

Export your API credentials and start the server with the TOML file:

export OPENROUTER_API_KEY="sk-..."
switchyard launch production --model production-escalation --config ./production_routes.toml

The server validates the route configuration at startup and surfaces any malformed judge parameters before accepting traffic.

Programmatic Configuration (Python)

For unit testing or custom client implementations, instantiate the stage router directly through the Python bindings defined in switchyard_rust/libsy.py:

from switchyard import algorithms

router = algorithms.stage_router(
    fast_router=algorithms.llm_task_classifier(
        model="openrouter/openai/gpt-4o-mini",
        temperature=0.1,
    ),
    strong_router=algorithms.llm_task_classifier(
        model="openrouter/openai/gpt-4o",
        temperature=0.0,
    ),
    escalation_judge_config={
        "judge": "openrouter/openai/gpt-4o-mini",
        "confirmations": 2,
        "max_output_tokens": 128,
        "escalation_note": "⚠️ Complex request—escalated to capable tier",
        "deescalation_note": "✅ Resolved—returning to efficient tier",
    },
)

This pattern enables rapid iteration on judge behavior in notebook environments before promoting configurations to production TOML files.

Complete Production TOML Example

# production_routes.toml

[routes.production_escalation]
id = "my-org/production-escalation"
mode = "escalation"

# Sub-router definitions

fast = { type = "llm_task_classifier", model = "openrouter/openai/gpt-4o-mini" }
strong = { type = "llm_task_classifier", model = "openrouter/openai/gpt-4o" }

# Escalation judge configuration

[escalation]
judge = "openrouter/openai/gpt-4o-mini"
confirmations = 2
max_output_tokens = 256
prompt = """Analyze the user request complexity.

Reply with exactly one word:
- ESCALATE if the request requires nuanced reasoning, multi-step logic, or specialized knowledge
- STAY if the request is straightforward, factual, or conversational"""

# Contextual handoff notes

escalation_note = "🚀 ESCALATED: Judge determined this request exceeds efficient tier capabilities."
deescalation_note = "🔁 DEEscalATED: Request resolved; efficient tier resumed."
only_on_wrong_signal_escalation = true

Runtime Execution Flow

According to the implementation in crates/libsy/src/algorithms/util/stage.rs, each request passes through this state machine:

1. Fast tier execution — The efficient model generates an initial response or partial result
2. Judge evaluation — The escalation judge receives the request context (and optionally the fast tier output) and renders a verdict
3. Policy validation — The system checks whether consecutive "escalate" signals meet the confirmations threshold
4. Promotion decision — If validated, the request forwards to the strong tier with escalation_note appended; otherwise, the fast tier result returns
5. De-escalation tracking — Successful strong tier completions may trigger demotion logic for subsequent related requests

The crates/libsy/src/algorithms/util/escalation.rs file implements the core verdict parsing, handling case normalization and strict output validation to prevent judge hallucinations from corrupting routing state.

Key Source Files Reference

File	Purpose
crates/switchyard-server/src/config.rs	TOML parsing and EscalationJudgeConfig construction
crates/libsy/src/algorithms/util/escalation.rs	Judge request construction, verdict logic, escalation policy enforcement
crates/libsy/src/algorithms/util/stage.rs	Stage router orchestration, handoff note injection, promotion gating
switchyard_rust/libsy.py	Python stage_router() binding with escalation-judge parameter support
docs/routing_algorithms/stage_router_routing.md	Official documentation for escalation routing patterns

Summary

• Switchyard's escalation router enables automatic model tier promotion based on runtime request complexity evaluation
• LLM judge configuration requires specifying a cheap evaluation model, confirmation threshold, and optional custom prompt
• Handoff notes preserve context across tier transitions without polluting non-escalated requests
• Higher confirmation values trade latency for promotion accuracy in noisy judge environments
• Both TOML and Python APIs support full configuration, enabling test-driven development of judge prompts

Frequently Asked Questions

What is the optimal confirmation threshold for the escalation judge?

Start with confirmations = 1 for latency-sensitive applications where occasional false promotions are acceptable. Increase to 2 or 3 when judge outputs show variability or when promotion costs significantly outweigh the penalty of under-routing. Monitor your specific judge model's precision-recall tradeoff through logged verdicts in crates/libsy/src/algorithms/util/escalation.rs.

Can the escalation judge access the efficient tier's output or only the raw request?

The judge receives configurable context through the prompt template. By default, implementations in crates/libsy/src/algorithms/util/escalation.rs can ingest both the original request and the fast tier's partial response. Customize this through the prompt parameter to expose whichever signals best predict promotion necessity for your workload.

Does Switchyard support de-escalation back to the efficient tier?

Yes. The deescalation_note parameter configures messages shown when requests return to the efficient tier, and the underlying stage router in crates/libsy/src/algorithms/util/stage.rs tracks promotion state across conversation turns. Automatic de-escalation policies depend on your specific strong router implementation and conversation boundary detection.

Which models work best as escalation judges?

Cost-optimized models with sub-200ms latency profiles excel as judges—gpt-4o-mini and equivalent small parameter models balance sufficient reasoning capacity with minimal evaluation overhead. Avoid using the same model for both judge and efficient tier to prevent correlated failure modes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}