# How Switchyard Manages Fallback When a Model Backend Becomes Unavailable

> Learn how Switchyard's multi-layered fallback strategy ensures model backend availability. Discover automatic retries and the First-Available algorithm with configurable routing policies.

- Repository: [NVIDIA-NeMo/Switchyard](https://github.com/NVIDIA-NeMo/Switchyard)
- Tags: how-to-guide
- Published: 2026-08-15

---

**Switchyard implements a multi-layered fallback strategy that automatically retries failed requests against alternate model backends when errors indicate unavailability or context-window exhaustion, using the First‑Available algorithm with configurable routing policies.**

Switchyard's routing layer is designed to keep LLM‑powered applications resilient against individual model provider failures. When a model backend cannot satisfy a request, Switchyard doesn't immediately surface the error to the user—instead, it classifies the failure and attempts the next candidate in the routing sequence. This article examines the implementation details of this fallback mechanism across the NVIDIA‑NeMo/Switchyard codebase.

## First‑Available Routing: The Core Fallback Loop

The primary fallback mechanism resides in the **First‑Available algorithm**, implemented in both Rust and Python interfaces. When a request is issued through a `Stage`, the system iterates through candidate model identifiers in order.

In [`crates/libsy/src/algorithms/stage.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy/src/algorithms/stage.rs), the routing logic coordinates this iteration. The underlying Rust implementation in [`crates/libsy-llm-client/src/run.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy-llm-client/src/run.rs) provides the `call_first_available` function:

```rust
// crates/libsy-llm-client/src/run.rs (excerpt)
async fn call_first_available(
    clients: &ClientRouter,
    call: &CallModel,
    ...
) -> Result<Response> {
    for (index, target) in call.models.iter().enumerate() {
        let request = request_for(&call.request, target);
        match call_one(clients, target, request, ...).await {
            Ok(resp) => return Ok(resp),
            Err(err) if index + 1 == call.models.len() => return Err(err),
            Err(err) => match fallback_reason(&err) {
                Some(reason) => tracing::info!(
                    from = %target,
                    to = %call.models[index + 1],
                    reason = reason.as_str(),
                    "model call failed; trying next candidate"
                ),
                None => return Err(err),
            },
        }
    }
    Err(LibsyError::NoTargets)
}

```

The loop terminates successfully on the first responsive backend, exhausts all candidates, or aborts early if an error is **not** classified as a fallback reason.

## Classifying Errors That Trigger Fallback

Not all errors warrant retrying another backend. The `fallback_reason` function in [`crates/libsy-llm-client/src/run.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy-llm-client/src/run.rs) performs precise classification of `LibsyError` variants into `RoutingFallbackReason` variants:

```rust
// crates/libsy-llm-client/src/run.rs
fn fallback_reason(error: &LibsyError) -> Option<RoutingFallbackReason> {
    match source {
        LlmClientError::ContextWindowExceeded { .. } => Some(RoutingFallbackReason::ContextWindow),
        LlmClientError::Transport { .. } | LlmClientError::Timeout { .. } => Some(RoutingFallbackReason::Unavailable),
        LlmClientError::UpstreamHttp { status, .. }
            if matches!(*status, 403 | 408 | 429) || (500..=599).contains(status) => Some(RoutingFallbackReason::Unavailable),
        _ => None,
    }
}

```

Two primary categories trigger fallback:

- **`Unavailable`** — transport failures, timeouts, or HTTP status codes indicating the backend cannot serve the request: 403 (Forbidden), 408 (Request Timeout), 429 (Too Many Requests), or any 5xx server error
- **`ContextWindow`** — the request exceeds the model's context window, prompting automatic downgrade to a model with sufficient capacity

Errors outside these categories—such as authentication failures or malformed requests—propagate immediately without retry.

## Python Interface: Configuring Fallback Behavior

The Python SDK exposes this mechanism through the `Stage` class with `LlmAlgorithm.FIRST_AVAILABLE`:

```python
from switchyard.libsy import Stage, LlmAlgorithm

# Define a stage that will try the models in order.

stage = Stage(
    algorithm=LlmAlgorithm.FIRST_AVAILABLE,
    models=[
        "openai/gpt-4o",          # Primary model

        "anthropic/claude-2",     # Fallback if the primary is unavailable

        "openrouter/llama-2-70b", # Final fallback

    ],
)

# The `run` call will automatically fall back on the next model

# whenever the current one returns an Unavailable or ContextWindow error.

response = stage.run(prompt="Explain quantum tunnelling in simple terms.")
print(response.text)

```

The `run()` method encapsulates the retry loop; callers receive the successful response without handling intermediate failures.

## Observability: Tracking Fallback Events

Every fallback event is recorded for operational visibility. The `RoutingFallbackStats` structure in [`crates/switchyard-server/src/stats/accumulator.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/src/stats/accumulator.rs) maintains counters for each reason type:

- `unavailable` — backend connectivity or capacity issues
- `context_window` — token limit exceeded

These statistics are exposed through the `/stats` HTTP endpoint and written to the routing log, enabling operators to identify problematic backends and tune fallback configurations.

## Proactive Health Detection for Local Proxies

CLI launchers embed lightweight health monitoring to preemptively detect unavailable backends. The `ProxyHealthMonitor` class in [`switchyard/cli/launchers/proxy_health_monitor.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard/cli/launchers/proxy_health_monitor.py) periodically probes locally‑run proxies:

```python

# switchyard/cli/launchers/proxy_health_monitor.py

class ProxyHealthMonitor:
    def __init__(self, port: int, interval: float = 5.0):
        self._url = f"http://127.0.0.1:{port}/health"
        self._healthy: bool | None = None
        self._interval = interval
        self._last_check = 0.0

    def poll(self):
        now = time.time()
        if now - self._last_check < self._interval:
            return
        self._last_check = now
        try:
            resp = httpx.get(self._url, timeout=2.0)
            self.__healthy = resp.status_code == 200
        except Exception:
            self._healthy = False

```

When `poll()` sets `self._healthy` to `False`, subsequent routing decisions treat that proxy as `Unavailable`, triggering the fallback path without waiting for a request to fail.

## Configuration Options for Fallback Control

The TOML configuration schema in [`crates/switchyard-server/src/config.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/src/config.rs) governs fallback behavior through several settings:

- Enable or disable fallback features entirely
- Configure `session_affinity` to maintain routing consistency across related requests
- Toggle `message_hash_fallback` for deterministic routing based on content

These options ensure that fallback remains predictable when multiple backends share a session or when reproducibility is required.

## Summary

Switchyard's fallback mechanism combines several techniques to maintain service availability:

- **Iterative retry loop** — `call_first_available` in [`crates/libsy-llm-client/src/run.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy-llm-client/src/run.rs) attempts candidates sequentially
- **Error classification** — `fallback_reason` distinguishes retriable from non‑retriable failures
- **Dual trigger conditions** — `Unavailable` (connectivity/capacity) and `ContextWindow` (token limits)
- **Observability integration** — `RoutingFallbackStats` tracks events in [`crates/switchyard-server/src/stats/accumulator.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/src/stats/accumulator.rs)
- **Proactive health checks** — `ProxyHealthMonitor` preemptively marks backends unhealthy
- **Configurable policies** — TOML settings in [`crates/switchyard-server/src/config.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/src/config.rs) control routing behavior

## Frequently Asked Questions

### What happens if all model backends in a stage are unavailable?

If every candidate in the models list fails with a classified fallback reason, `call_first_available` returns the final error and propagates it to the caller. The Python `Stage.run()` will raise an exception, while the Rust function returns `Err(LibsyError::NoTargets)` only if the list is empty initially.

### How does Switchyard handle rate limiting (429 errors)?

HTTP 429 status codes are explicitly matched in `fallback_reason` as `RoutingFallbackReason::Unavailable`, causing automatic retry against the next backend. This prevents a single rate‑limited provider from stalling the entire request.

### Can fallback be disabled for specific requests?

Yes. The TOML configuration in [`crates/switchyard-server/src/config.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/src/config.rs) allows disabling fallback features globally, and `session_affinity` settings can enforce sticky routing that bypasses the First‑Available algorithm for sessions requiring consistency.