How to Implement Health-Aware Fallback with Automatic Endpoint Failover in Switchyard

Switchyard implements health-aware fallback by combining continuous health probes, a scoring algorithm, and a routing layer that automatically redirects traffic to the highest-scoring healthy endpoint.

Switchyard's routing architecture centers on the libsy crate, which provides intelligent LLM backend selection based on real-time health metrics. By configuring health polling and failover thresholds, you can build resilient deployments that survive individual endpoint failures without manual intervention. This guide walks through the complete implementation using the actual Switchyard source code from NVIDIA-NeMo/Switchyard.

Understanding the Health-Aware Routing Architecture

Switchyard separates health monitoring from request handling across three layers:

• Router core (crates/libsy/src/router.rs) — Implements routing algorithms including RoundRobin, Weighted, and health_aware
• Health monitoring (crates/switchyard-server/src/health.rs) — Aggregates per-endpoint health metrics into Prometheus-compatible endpoints
• Configuration layer (switchyard_rust/libsy.py, CLI launchers) — Exposes Rust functionality to Python and parses deployment TOML files

The health-aware router continuously polls configured endpoints and maintains an internal health score for each. When routing requests, it selects the endpoint with the highest score that exceeds the minimum threshold.

How Health Scoring and Failover Work

Health Probe Format

Each LLM endpoint must expose a health service (default port 4001) returning JSON:

{
  "available": true,
  "latency_ms": 45,
  "error_rate": 0.02
}

The Switchyard server aggregates these into Prometheus metrics: switchyard_endpoint_up, switchyard_endpoint_latency_ms, and switchyard_endpoint_error_rate.

Score Calculation

In crates/libsy/src/router.rs, the health score computes as:

score = 1 / (latency_ms × (1 + error_rate))

Scores refresh every health_poll_interval (default 5 seconds). Higher scores indicate healthier endpoints.

Failover Logic

When processing a request, the router:

1. Filters endpoints with score >= health_score_min
2. Selects the maximum-scoring endpoint
3. If the previously-selected endpoint drops below threshold, automatically switches to the next-best candidate
4. Re-evaluates on every poll cycle, enabling automatic recovery when primary endpoints heal

Implementing Health-Aware Fallback via TOML Configuration

The simplest approach uses a deployment configuration file passed to the Switchyard CLI.

Step 1: Create the Deployment File

# routes.toml

[[endpoint]]
name = "primary"
url = "http://localhost:4000/v1/chat"
health_url = "http://localhost:4001/health"

[[endpoint]]
name = "backup"
url = "http://localhost:5000/v1/chat"
health_url = "http://localhost:5001/health"

[router]
type = "health_aware"
health_poll_interval = "5s"
health_score_min = 0.1

Key parameters:

• health_url — Required for health-aware routing; omitting falls back to simple round-robin
• health_poll_interval — Duration string parsed by humantime in the Rust core
• health_score_min — Minimum viable score; endpoints scoring below are excluded from selection

Step 2: Start the Native Server

switchyard-server --config routes.toml --port 4000

The server validates the TOML, initializes the HealthAwareRouter struct from crates/libsy/src/router.rs, and spawns the health polling background task.

Step 3: Launch Client Applications

switchyard launch claude --model my-route --config routes.toml

The launcher in switchyard/cli/launch_command.py reads the same TOML, ensuring router configuration stays synchronized between server and client.

Programmatic Implementation Using Python

For custom integrations, instantiate LibSyRouter directly via the Python façade:

from switchyard_rust import LibSyRouter, EndpointConfig

# Configure endpoints with health monitoring endpoints.

primary = EndpointConfig(
    name="primary",
    url="http://localhost:4000/v1/chat",
    health_url="http://localhost:4001/health",
)

backup = EndpointConfig(
    name="backup",
    url="http://localhost:5000/v1/chat",
    health_url="http://localhost:5001/health",
)

# Initialize health-aware router with explicit parameters.

router = LibSyRouter(
    endpoints=[primary, backup],
    router_type="health_aware",
    health_poll_interval=5.0,   # seconds, passed to Rust as Duration

    health_score_min=0.1,
)

# Route automatically selects healthiest endpoint.

response = router.route_chat(
    messages=[{"role": "user", "content": "Hello"}],
    model="gpt-4"
)

The Python wrapper in switchyard_rust/libsy.py uses PyO3 to expose the underlying Rust RouterHandle, which manages the background health polling task transparently.

Customizing Health Endpoint Behavior

If your LLM backend lacks a native health endpoint, wrap it with a minimal health server:

from fastapi import FastAPI
import httpx

app = FastAPI()
TARGET_LLM = "http://localhost:8000/v1/chat"

@app.get("/health")
async def health():
    async with httpx.AsyncClient() as client:
        start = time.time()
        try:
            # Probe actual LLM with minimal request

            await client.post(TARGET_LLM, json={
                "model": "gpt-4",
                "messages": [{"role": "user", "content": "hi"}],
                "max_tokens": 1
            }, timeout=10.0)
            latency = (time.time() - start) * 1000
            return {
                "available": True,
                "latency_ms": latency,
                "error_rate": 0.0
            }
        except Exception:
            return {
                "available": False,
                "latency_ms": 99999,
                "error_rate": 1.0
            }

Deploy this alongside your LLM instance and reference it as health_url in your Switchyard configuration.

Monitoring Failover Events

The Switchyard server exports detailed metrics for observability. Query the metrics endpoint:

curl http://localhost:4000/metrics | grep switchyard_endpoint

Relevant metrics defined in crates/switchyard-server/src/health.rs:

• switchyard_endpoint_up{endpoint="primary"} — 1 if health probe succeeds, 0 otherwise
• switchyard_endpoint_latency_ms{endpoint="primary"} — Last measured latency
• switchyard_endpoint_score{endpoint="primary"} — Computed routing score

Set up alerts on switchyard_endpoint_up == 0 for notification, though Switchyard handles the actual failover automatically.

Tuning Failover Sensitivity

Adjust these parameters based on your reliability requirements:

Parameter	Lower Value	Higher Value
health_poll_interval	Faster detection, more load	Slower detection, less overhead
health_score_min	More aggressive failover	Permits degraded endpoints

For latency-sensitive applications, combine health_aware routing with Weighted configuration to prefer faster endpoints under normal conditions while maintaining fallback coverage.

Summary

• Health-aware fallback in Switchyard requires configuring health_url for each endpoint and setting router.type = "health_aware"
• The health score formula (1 / (latency × (1 + error_rate))) balances responsiveness against error rates
• Automatic recovery occurs without intervention when failed endpoints heal
• Implementation options include TOML configuration for CLI workflows or programmatic LibSyRouter for custom integrations
• Key source files: crates/libsy/src/router.rs (routing logic), crates/switchyard-server/src/health.rs (health aggregation), switchyard_rust/libsy.py (Python bindings)

Frequently Asked Questions

What happens if all endpoints fall below the health score threshold?

Router behavior depends on implementation details in crates/libsy/src/router.rs. Typically, if no endpoint exceeds health_score_min, the router selects the highest-scoring available endpoint regardless of threshold, degraded service being preferable to complete outage. Monitor switchyard_endpoint_score metrics to detect this scenario.

Can I use health-aware routing with more than two endpoints?

Yes. The health-aware router in crates/libsy/src/router.rs maintains a scored priority queue of all configured endpoints. Configure any number of endpoints in your TOML [[endpoint]] array; the router automatically distributes traffic across all healthy candidates, preferring higher scores.

How does health-aware routing interact with weighted routing?

As of the current Switchyard implementation, router.type selects a single algorithm. For combined weighted and health-aware behavior, extend the router configuration or implement custom logic in crates/libsy/src/router.rs that multiplies health scores by configured weights. The modular Router trait supports such extensions.

Does the Python LibSyRouter require a running switchyard-server process?

No. When instantiated directly, LibSyRouter from switchyard_rust/libsy.py embeds the Rust router as a native extension, spawning its own health polling threads. However, using switchyard launch with TOML configuration requires the server process for full metric export and multi-client coordination.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}