How Switchyard Manages Fallback When a Model Backend Becomes Unavailable

Switchyard implements a multi-layered fallback strategy that automatically retries failed requests against alternate model backends when errors indicate unavailability or context-window exhaustion, using the First‑Available algorithm with configurable routing policies.

Switchyard's routing layer is designed to keep LLM‑powered applications resilient against individual model provider failures. When a model backend cannot satisfy a request, Switchyard doesn't immediately surface the error to the user—instead, it classifies the failure and attempts the next candidate in the routing sequence. This article examines the implementation details of this fallback mechanism across the NVIDIA‑NeMo/Switchyard codebase.

First‑Available Routing: The Core Fallback Loop

The primary fallback mechanism resides in the First‑Available algorithm, implemented in both Rust and Python interfaces. When a request is issued through a Stage, the system iterates through candidate model identifiers in order.

In crates/libsy/src/algorithms/stage.rs, the routing logic coordinates this iteration. The underlying Rust implementation in crates/libsy-llm-client/src/run.rs provides the call_first_available function:

// crates/libsy-llm-client/src/run.rs (excerpt)
async fn call_first_available(
    clients: &ClientRouter,
    call: &CallModel,
    ...
) -> Result<Response> {
    for (index, target) in call.models.iter().enumerate() {
        let request = request_for(&call.request, target);
        match call_one(clients, target, request, ...).await {
            Ok(resp) => return Ok(resp),
            Err(err) if index + 1 == call.models.len() => return Err(err),
            Err(err) => match fallback_reason(&err) {
                Some(reason) => tracing::info!(
                    from = %target,
                    to = %call.models[index + 1],
                    reason = reason.as_str(),
                    "model call failed; trying next candidate"
                ),
                None => return Err(err),
            },
        }
    }
    Err(LibsyError::NoTargets)
}

The loop terminates successfully on the first responsive backend, exhausts all candidates, or aborts early if an error is not classified as a fallback reason.

Classifying Errors That Trigger Fallback

Not all errors warrant retrying another backend. The fallback_reason function in crates/libsy-llm-client/src/run.rs performs precise classification of LibsyError variants into RoutingFallbackReason variants:

// crates/libsy-llm-client/src/run.rs
fn fallback_reason(error: &LibsyError) -> Option<RoutingFallbackReason> {
    match source {
        LlmClientError::ContextWindowExceeded { .. } => Some(RoutingFallbackReason::ContextWindow),
        LlmClientError::Transport { .. } | LlmClientError::Timeout { .. } => Some(RoutingFallbackReason::Unavailable),
        LlmClientError::UpstreamHttp { status, .. }
            if matches!(*status, 403 | 408 | 429) || (500..=599).contains(status) => Some(RoutingFallbackReason::Unavailable),
        _ => None,
    }
}

Two primary categories trigger fallback:

• Unavailable — transport failures, timeouts, or HTTP status codes indicating the backend cannot serve the request: 403 (Forbidden), 408 (Request Timeout), 429 (Too Many Requests), or any 5xx server error
• ContextWindow — the request exceeds the model's context window, prompting automatic downgrade to a model with sufficient capacity

Errors outside these categories—such as authentication failures or malformed requests—propagate immediately without retry.

Python Interface: Configuring Fallback Behavior

The Python SDK exposes this mechanism through the Stage class with LlmAlgorithm.FIRST_AVAILABLE:

from switchyard.libsy import Stage, LlmAlgorithm

# Define a stage that will try the models in order.

stage = Stage(
    algorithm=LlmAlgorithm.FIRST_AVAILABLE,
    models=[
        "openai/gpt-4o",          # Primary model

        "anthropic/claude-2",     # Fallback if the primary is unavailable

        "openrouter/llama-2-70b", # Final fallback

    ],
)

# The `run` call will automatically fall back on the next model

# whenever the current one returns an Unavailable or ContextWindow error.

response = stage.run(prompt="Explain quantum tunnelling in simple terms.")
print(response.text)

The run() method encapsulates the retry loop; callers receive the successful response without handling intermediate failures.

Observability: Tracking Fallback Events

Every fallback event is recorded for operational visibility. The RoutingFallbackStats structure in crates/switchyard-server/src/stats/accumulator.rs maintains counters for each reason type:

• unavailable — backend connectivity or capacity issues
• context_window — token limit exceeded

These statistics are exposed through the /stats HTTP endpoint and written to the routing log, enabling operators to identify problematic backends and tune fallback configurations.

Proactive Health Detection for Local Proxies

CLI launchers embed lightweight health monitoring to preemptively detect unavailable backends. The ProxyHealthMonitor class in switchyard/cli/launchers/proxy_health_monitor.py periodically probes locally‑run proxies:

# switchyard/cli/launchers/proxy_health_monitor.py

class ProxyHealthMonitor:
    def __init__(self, port: int, interval: float = 5.0):
        self._url = f"http://127.0.0.1:{port}/health"
        self._healthy: bool | None = None
        self._interval = interval
        self._last_check = 0.0

    def poll(self):
        now = time.time()
        if now - self._last_check < self._interval:
            return
        self._last_check = now
        try:
            resp = httpx.get(self._url, timeout=2.0)
            self.__healthy = resp.status_code == 200
        except Exception:
            self._healthy = False

When poll() sets self._healthy to False, subsequent routing decisions treat that proxy as Unavailable, triggering the fallback path without waiting for a request to fail.

Configuration Options for Fallback Control

The TOML configuration schema in crates/switchyard-server/src/config.rs governs fallback behavior through several settings:

• Enable or disable fallback features entirely
• Configure session_affinity to maintain routing consistency across related requests
• Toggle message_hash_fallback for deterministic routing based on content

These options ensure that fallback remains predictable when multiple backends share a session or when reproducibility is required.

Summary

Switchyard's fallback mechanism combines several techniques to maintain service availability:

• Iterative retry loop — call_first_available in crates/libsy-llm-client/src/run.rs attempts candidates sequentially
• Error classification — fallback_reason distinguishes retriable from non‑retriable failures
• Dual trigger conditions — Unavailable (connectivity/capacity) and ContextWindow (token limits)
• Observability integration — RoutingFallbackStats tracks events in crates/switchyard-server/src/stats/accumulator.rs
• Proactive health checks — ProxyHealthMonitor preemptively marks backends unhealthy
• Configurable policies — TOML settings in crates/switchyard-server/src/config.rs control routing behavior

Frequently Asked Questions

What happens if all model backends in a stage are unavailable?

If every candidate in the models list fails with a classified fallback reason, call_first_available returns the final error and propagates it to the caller. The Python Stage.run() will raise an exception, while the Rust function returns Err(LibsyError::NoTargets) only if the list is empty initially.

How does Switchyard handle rate limiting (429 errors)?

HTTP 429 status codes are explicitly matched in fallback_reason as RoutingFallbackReason::Unavailable, causing automatic retry against the next backend. This prevents a single rate‑limited provider from stalling the entire request.

Can fallback be disabled for specific requests?

Yes. The TOML configuration in crates/switchyard-server/src/config.rs allows disabling fallback features globally, and session_affinity settings can enforce sticky routing that bypasses the First‑Available algorithm for sessions requiring consistency.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}