# How to Implement Health-Aware Fallback with Automatic Endpoint Failover in Switchyard

> Implement health-aware fallback and automatic endpoint failover in Switchyard. Learn how continuous health probes, scoring, and routing ensure traffic always goes to the best available endpoint.

- Repository: [NVIDIA-NeMo/Switchyard](https://github.com/NVIDIA-NeMo/Switchyard)
- Tags: how-to-guide
- Published: 2026-08-15

---

**Switchyard implements health-aware fallback by combining continuous health probes, a scoring algorithm, and a routing layer that automatically redirects traffic to the highest-scoring healthy endpoint.**

Switchyard's routing architecture centers on the `libsy` crate, which provides intelligent LLM backend selection based on real-time health metrics. By configuring health polling and failover thresholds, you can build resilient deployments that survive individual endpoint failures without manual intervention. This guide walks through the complete implementation using the actual Switchyard source code from NVIDIA-NeMo/Switchyard.

## Understanding the Health-Aware Routing Architecture

Switchyard separates health monitoring from request handling across three layers:

- **Router core** ([`crates/libsy/src/router.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy/src/router.rs)) — Implements routing algorithms including `RoundRobin`, `Weighted`, and `health_aware`
- **Health monitoring** ([`crates/switchyard-server/src/health.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/src/health.rs)) — Aggregates per-endpoint health metrics into Prometheus-compatible endpoints
- **Configuration layer** ([`switchyard_rust/libsy.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard_rust/libsy.py), CLI launchers) — Exposes Rust functionality to Python and parses deployment TOML files

The health-aware router continuously polls configured endpoints and maintains an internal **health score** for each. When routing requests, it selects the endpoint with the highest score that exceeds the minimum threshold.

## How Health Scoring and Failover Work

### Health Probe Format

Each LLM endpoint must expose a health service (default port `4001`) returning JSON:

```json
{
  "available": true,
  "latency_ms": 45,
  "error_rate": 0.02
}

```

The Switchyard server aggregates these into Prometheus metrics: `switchyard_endpoint_up`, `switchyard_endpoint_latency_ms`, and `switchyard_endpoint_error_rate`.

### Score Calculation

In [`crates/libsy/src/router.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy/src/router.rs), the health score computes as:

```

score = 1 / (latency_ms × (1 + error_rate))

```

Scores refresh every `health_poll_interval` (default 5 seconds). Higher scores indicate healthier endpoints.

### Failover Logic

When processing a request, the router:

1. Filters endpoints with `score >= health_score_min`
2. Selects the maximum-scoring endpoint
3. If the previously-selected endpoint drops below threshold, automatically switches to the next-best candidate
4. Re-evaluates on every poll cycle, enabling automatic recovery when primary endpoints heal

## Implementing Health-Aware Fallback via TOML Configuration

The simplest approach uses a deployment configuration file passed to the Switchyard CLI.

### Step 1: Create the Deployment File

```toml

# routes.toml

[[endpoint]]
name = "primary"
url = "http://localhost:4000/v1/chat"
health_url = "http://localhost:4001/health"

[[endpoint]]
name = "backup"
url = "http://localhost:5000/v1/chat"
health_url = "http://localhost:5001/health"

[router]
type = "health_aware"
health_poll_interval = "5s"
health_score_min = 0.1

```

Key parameters:
- `health_url` — Required for health-aware routing; omitting falls back to simple round-robin
- `health_poll_interval` — Duration string parsed by `humantime` in the Rust core
- `health_score_min` — Minimum viable score; endpoints scoring below are excluded from selection

### Step 2: Start the Native Server

```bash
switchyard-server --config routes.toml --port 4000

```

The server validates the TOML, initializes the `HealthAwareRouter` struct from [`crates/libsy/src/router.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy/src/router.rs), and spawns the health polling background task.

### Step 3: Launch Client Applications

```bash
switchyard launch claude --model my-route --config routes.toml

```

The launcher in [`switchyard/cli/launch_command.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard/cli/launch_command.py) reads the same TOML, ensuring router configuration stays synchronized between server and client.

## Programmatic Implementation Using Python

For custom integrations, instantiate `LibSyRouter` directly via the Python façade:

```python
from switchyard_rust import LibSyRouter, EndpointConfig

# Configure endpoints with health monitoring endpoints.

primary = EndpointConfig(
    name="primary",
    url="http://localhost:4000/v1/chat",
    health_url="http://localhost:4001/health",
)

backup = EndpointConfig(
    name="backup",
    url="http://localhost:5000/v1/chat",
    health_url="http://localhost:5001/health",
)

# Initialize health-aware router with explicit parameters.

router = LibSyRouter(
    endpoints=[primary, backup],
    router_type="health_aware",
    health_poll_interval=5.0,   # seconds, passed to Rust as Duration

    health_score_min=0.1,
)

# Route automatically selects healthiest endpoint.

response = router.route_chat(
    messages=[{"role": "user", "content": "Hello"}],
    model="gpt-4"
)

```

The Python wrapper in [`switchyard_rust/libsy.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard_rust/libsy.py) uses PyO3 to expose the underlying Rust `RouterHandle`, which manages the background health polling task transparently.

## Customizing Health Endpoint Behavior

If your LLM backend lacks a native health endpoint, wrap it with a minimal health server:

```python
from fastapi import FastAPI
import httpx

app = FastAPI()
TARGET_LLM = "http://localhost:8000/v1/chat"

@app.get("/health")
async def health():
    async with httpx.AsyncClient() as client:
        start = time.time()
        try:
            # Probe actual LLM with minimal request

            await client.post(TARGET_LLM, json={
                "model": "gpt-4",
                "messages": [{"role": "user", "content": "hi"}],
                "max_tokens": 1
            }, timeout=10.0)
            latency = (time.time() - start) * 1000
            return {
                "available": True,
                "latency_ms": latency,
                "error_rate": 0.0
            }
        except Exception:
            return {
                "available": False,
                "latency_ms": 99999,
                "error_rate": 1.0
            }

```

Deploy this alongside your LLM instance and reference it as `health_url` in your Switchyard configuration.

## Monitoring Failover Events

The Switchyard server exports detailed metrics for observability. Query the metrics endpoint:

```bash
curl http://localhost:4000/metrics | grep switchyard_endpoint

```

Relevant metrics defined in [`crates/switchyard-server/src/health.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/src/health.rs):

- `switchyard_endpoint_up{endpoint="primary"}` — 1 if health probe succeeds, 0 otherwise
- `switchyard_endpoint_latency_ms{endpoint="primary"}` — Last measured latency
- `switchyard_endpoint_score{endpoint="primary"}` — Computed routing score

Set up alerts on `switchyard_endpoint_up == 0` for notification, though Switchyard handles the actual failover automatically.

## Tuning Failover Sensitivity

Adjust these parameters based on your reliability requirements:

| Parameter | Lower Value | Higher Value |
|-----------|-------------|--------------|
| `health_poll_interval` | Faster detection, more load | Slower detection, less overhead |
| `health_score_min` | More aggressive failover | Permits degraded endpoints |

For latency-sensitive applications, combine `health_aware` routing with `Weighted` configuration to prefer faster endpoints under normal conditions while maintaining fallback coverage.

## Summary

- **Health-aware fallback** in Switchyard requires configuring `health_url` for each endpoint and setting `router.type = "health_aware"`
- The **health score formula** (`1 / (latency × (1 + error_rate))`) balances responsiveness against error rates
- **Automatic recovery** occurs without intervention when failed endpoints heal
- Implementation options include **TOML configuration** for CLI workflows or **programmatic `LibSyRouter`** for custom integrations
- Key source files: [`crates/libsy/src/router.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy/src/router.rs) (routing logic), [`crates/switchyard-server/src/health.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/src/health.rs) (health aggregation), [`switchyard_rust/libsy.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard_rust/libsy.py) (Python bindings)

## Frequently Asked Questions

### What happens if all endpoints fall below the health score threshold?

Router behavior depends on implementation details in [`crates/libsy/src/router.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy/src/router.rs). Typically, if no endpoint exceeds `health_score_min`, the router selects the highest-scoring available endpoint regardless of threshold, degraded service being preferable to complete outage. Monitor `switchyard_endpoint_score` metrics to detect this scenario.

### Can I use health-aware routing with more than two endpoints?

Yes. The health-aware router in [`crates/libsy/src/router.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy/src/router.rs) maintains a scored priority queue of all configured endpoints. Configure any number of endpoints in your TOML `[[endpoint]]` array; the router automatically distributes traffic across all healthy candidates, preferring higher scores.

### How does health-aware routing interact with weighted routing?

As of the current Switchyard implementation, `router.type` selects a single algorithm. For combined weighted and health-aware behavior, extend the router configuration or implement custom logic in [`crates/libsy/src/router.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy/src/router.rs) that multiplies health scores by configured weights. The modular `Router` trait supports such extensions.

### Does the Python `LibSyRouter` require a running switchyard-server process?

No. When instantiated directly, `LibSyRouter` from [`switchyard_rust/libsy.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard_rust/libsy.py) embeds the Rust router as a native extension, spawning its own health polling threads. However, using `switchyard launch` with TOML configuration requires the server process for full metric export and multi-client coordination.