# How to Configure an Escalation Router with an LLM Judge for Automatic Model Tier Promotion in Switchyard

> Configure an escalation router with an LLM judge in Switchyard to automatically promote model tiers. Enable efficient models to hand off requests to capable models when needed.

- Repository: [NVIDIA-NeMo/Switchyard](https://github.com/NVIDIA-NeMo/Switchyard)
- Tags: how-to-guide
- Published: 2026-08-15

---

**Switchyard's escalation router lets a low-cost "efficient" tier model automatically hand off requests to a higher-cost "capable" tier when an LLM-based escalation judge decides the request exceeds the efficient model's capabilities.**

Switchyard's **escalation router** is a cost-optimization pattern that routes easy requests to cheap models and promotes hard requests to capable models based on runtime evaluation. By configuring an **LLM judge** as the promotion gatekeeper, you avoid wasting expensive model calls on trivial tasks while ensuring complex queries receive adequate compute.

## Architecture of the Escalation Router

The escalation system comprises four coordinated components:

| Component | Role | Source Location |
|-----------|------|-----------------|
| **Stage router** (`stage_router`) | Orchestrates the two-tier routing decision between fast and strong sub-routers | [`switchyard_rust/libsy.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard_rust/libsy.py) |
| **Escalation judge** | Lightweight LLM that renders *escalate* or *stay* verdicts on each request | [`crates/libsy/src/algorithms/util/escalation.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy/src/algorithms/util/escalation.rs) |
| **Escalation policy** | Defines how many consecutive "escalate" signals trigger actual promotion | [`crates/libsy/src/algorithms/util/escalation.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy/src/algorithms/util/escalation.rs) |
| **Handoff notes** | Optional contextual strings passed between tiers during promotion/demotion | [`crates/libsy/src/algorithms/util/stage.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy/src/algorithms/util/stage.rs) |

The execution flow follows a strict sequence: the **fast router** processes the request first, the **escalation judge** evaluates whether promotion is warranted, and—if the policy threshold is met—the **strong router** completes the response. This architecture separates routing orchestration from evaluation logic, allowing independent iteration on judge prompts and promotion policies.

## Step-by-Step Configuration

### 1. Define the Escalation Route in TOML

Create a deployment configuration file (e.g., [`production_routes.toml`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/production_routes.toml)) that declares an escalation-mode route with both sub-routers specified:

```toml
[routes.production_escalation]
id = "my-org/production-escalation"
mode = "escalation"

fast = { type = "random", name = "gpt-4o-mini" }
strong = { type = "llm_task_classifier", model = "openrouter/openai/gpt-4o" }

```

The `mode = "escalation"` directive activates the stage router with promotion logic enabled.

### 2. Configure the LLM Judge Parameters

Within the same route block, add the `escalation` table with judge-specific settings:

```toml
[routes.production_escalation.escalation]
judge = "openrouter/openai/gpt-4o-mini"
confirmations = 2
max_output_tokens = 256
prompt = "Evaluate if this request requires advanced reasoning. Reply ESCALATE or STAY."

```

Critical parameters include:

- **`judge`**: Model identifier for the evaluation LLM—typically a cheap, fast model like `gpt-4o-mini`
- **`confirmations`**: Consecutive "escalate" verdicts required before actual promotion (default: 1)
- **`max_output_tokens`**: Hard limit on judge response length
- **`prompt`**: Optional override for the built-in escalation evaluation prompt

Higher `confirmations` values reduce false promotions caused by judge noise but increase latency for legitimately hard requests.

### 3. Add Handoff Notes for Context Preservation

Configure notes that inform each tier when promotions or demotions occur:

```toml
escalation_note = "🚀 PROMOTED: This request was escalated from the efficient tier due to complexity."
deescalation_note = "🔁 RETURNED: Request completed; resuming efficient tier for follow-ups."
only_on_wrong_signal_escalation = true

```

When `only_on_wrong_signal_escalation = true` (the default), notes attach only to signal-driven escalations rather than all routing decisions. This prevents note pollution during manual tier selection.

### 4. Launch Switchyard with the Custom Configuration

Export your API credentials and start the server with the TOML file:

```bash
export OPENROUTER_API_KEY="sk-..."
switchyard launch production --model production-escalation --config ./production_routes.toml

```

The server validates the route configuration at startup and surfaces any malformed judge parameters before accepting traffic.

## Programmatic Configuration (Python)

For unit testing or custom client implementations, instantiate the stage router directly through the Python bindings defined in [`switchyard_rust/libsy.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard_rust/libsy.py):

```python
from switchyard import algorithms

router = algorithms.stage_router(
    fast_router=algorithms.llm_task_classifier(
        model="openrouter/openai/gpt-4o-mini",
        temperature=0.1,
    ),
    strong_router=algorithms.llm_task_classifier(
        model="openrouter/openai/gpt-4o",
        temperature=0.0,
    ),
    escalation_judge_config={
        "judge": "openrouter/openai/gpt-4o-mini",
        "confirmations": 2,
        "max_output_tokens": 128,
        "escalation_note": "⚠️ Complex request—escalated to capable tier",
        "deescalation_note": "✅ Resolved—returning to efficient tier",
    },
)

```

This pattern enables rapid iteration on judge behavior in notebook environments before promoting configurations to production TOML files.

## Complete Production TOML Example

```toml

# production_routes.toml

[routes.production_escalation]
id = "my-org/production-escalation"
mode = "escalation"

# Sub-router definitions

fast = { type = "llm_task_classifier", model = "openrouter/openai/gpt-4o-mini" }
strong = { type = "llm_task_classifier", model = "openrouter/openai/gpt-4o" }

# Escalation judge configuration

[escalation]
judge = "openrouter/openai/gpt-4o-mini"
confirmations = 2
max_output_tokens = 256
prompt = """Analyze the user request complexity.

Reply with exactly one word:
- ESCALATE if the request requires nuanced reasoning, multi-step logic, or specialized knowledge
- STAY if the request is straightforward, factual, or conversational"""

# Contextual handoff notes

escalation_note = "🚀 ESCALATED: Judge determined this request exceeds efficient tier capabilities."
deescalation_note = "🔁 DEEscalATED: Request resolved; efficient tier resumed."
only_on_wrong_signal_escalation = true

```

## Runtime Execution Flow

According to the implementation in [`crates/libsy/src/algorithms/util/stage.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy/src/algorithms/util/stage.rs), each request passes through this state machine:

1. **Fast tier execution** — The efficient model generates an initial response or partial result
2. **Judge evaluation** — The escalation judge receives the request context (and optionally the fast tier output) and renders a verdict
3. **Policy validation** — The system checks whether consecutive "escalate" signals meet the `confirmations` threshold
4. **Promotion decision** — If validated, the request forwards to the strong tier with `escalation_note` appended; otherwise, the fast tier result returns
5. **De-escalation tracking** — Successful strong tier completions may trigger demotion logic for subsequent related requests

The [`crates/libsy/src/algorithms/util/escalation.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy/src/algorithms/util/escalation.rs) file implements the core verdict parsing, handling case normalization and strict output validation to prevent judge hallucinations from corrupting routing state.

## Key Source Files Reference

| File | Purpose |
|------|---------|
| [`crates/switchyard-server/src/config.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/src/config.rs) | TOML parsing and `EscalationJudgeConfig` construction |
| [`crates/libsy/src/algorithms/util/escalation.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy/src/algorithms/util/escalation.rs) | Judge request construction, verdict logic, escalation policy enforcement |
| [`crates/libsy/src/algorithms/util/stage.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy/src/algorithms/util/stage.rs) | Stage router orchestration, handoff note injection, promotion gating |
| [`switchyard_rust/libsy.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard_rust/libsy.py) | Python `stage_router()` binding with escalation-judge parameter support |
| [`docs/routing_algorithms/stage_router_routing.md`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/docs/routing_algorithms/stage_router_routing.md) | Official documentation for escalation routing patterns |

## Summary

- **Switchyard's escalation router** enables automatic model tier promotion based on runtime request complexity evaluation
- **LLM judge configuration** requires specifying a cheap evaluation model, confirmation threshold, and optional custom prompt
- **Handoff notes** preserve context across tier transitions without polluting non-escalated requests
- **Higher confirmation values** trade latency for promotion accuracy in noisy judge environments
- **Both TOML and Python APIs** support full configuration, enabling test-driven development of judge prompts

## Frequently Asked Questions

### What is the optimal confirmation threshold for the escalation judge?

Start with `confirmations = 1` for latency-sensitive applications where occasional false promotions are acceptable. Increase to `2` or `3` when judge outputs show variability or when promotion costs significantly outweigh the penalty of under-routing. Monitor your specific judge model's precision-recall tradeoff through logged verdicts in [`crates/libsy/src/algorithms/util/escalation.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy/src/algorithms/util/escalation.rs).

### Can the escalation judge access the efficient tier's output or only the raw request?

The judge receives configurable context through the prompt template. By default, implementations in [`crates/libsy/src/algorithms/util/escalation.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy/src/algorithms/util/escalation.rs) can ingest both the original request and the fast tier's partial response. Customize this through the `prompt` parameter to expose whichever signals best predict promotion necessity for your workload.

### Does Switchyard support de-escalation back to the efficient tier?

Yes. The `deescalation_note` parameter configures messages shown when requests return to the efficient tier, and the underlying stage router in [`crates/libsy/src/algorithms/util/stage.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy/src/algorithms/util/stage.rs) tracks promotion state across conversation turns. Automatic de-escalation policies depend on your specific strong router implementation and conversation boundary detection.

### Which models work best as escalation judges?

Cost-optimized models with sub-200ms latency profiles excel as judges—`gpt-4o-mini` and equivalent small parameter models balance sufficient reasoning capacity with minimal evaluation overhead. Avoid using the same model for both judge and efficient tier to prevent correlated failure modes.