How Switchyard Manages Concurrent Requests Across Different Routing Algorithms

Switchyard manages concurrent requests using Rust's Tokio async runtime with a per-request driver/channel architecture that isolates each client request while allowing routing algorithms to run in parallel.

Switchyard, NVIDIA's open-source LLM routing framework, implements a sophisticated concurrency model that enables multiple routing algorithms to process requests simultaneously without interference. The system achieves this through carefully designed isolation primitives, bounded channels for back-pressure, and Tokio's lightweight task scheduling.

The Per-Request Driver Architecture

At the core of Switchyard's concurrency model is the per-request driver pattern. When a request enters the system, the server creates a Driver instance that owns a dedicated MPSC channel (step_tx) with a fixed capacity of 1.

In [crates/libsy/src/algorithm.rs](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy/src/algorithm.rs), the Algorithm::run_stream method receives this driver and produces a StepStream:

// Conceptual structure from algorithm.rs
pub trait Algorithm {
    async fn run_stream(&self, driver: Driver) -> StepStream;
}

The channel's capacity of 1 provides natural back-pressure: the routing algorithm can only emit the next step when the consumer (the host performing the actual model call) has processed the previous one. This prevents unbounded memory growth and ensures predictable resource usage.

Each request receives its own driver and step receiver, so no state is shared between requests. This isolation is fundamental to safe concurrent execution across different routing algorithms.

Algorithm Execution and Step Streaming

Routing algorithms implement the Algorithm trait and run inside dedicated async tasks. The run_stream method yields a stream of Step items that drive the request lifecycle:

• Step::CallModel – Triggers a model call fulfilled by the host via libsy-llm-client::run
• Step::Decision – Records a routing decision, optionally sent back to the client
• Step::Done – Signals the final response

Because each Step is awaited asynchronously, multiple model calls can be in flight simultaneously for the same request. This enables advanced patterns like the stage router's fallback behavior, where a second candidate model is queried if the first fails.

The stage router implementation in [crates/libsy/src/algorithms/stage.rs](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy/src/algorithms/stage.rs) demonstrates this pattern, while the random router in [crates/libsy/src/algorithms/rand.rs](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy/src/algorithms/rand.rs) shows the simplest case.

Server-Side Concurrency with Tokio and Axum

The native server (switchyard-server) leverages Axum and Tokio for high-throughput request handling. In [crates/switchyard-server/src/main.rs](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/src/main.rs), the implementation follows this pattern:

// From switchyard-server/src/main.rs
let listener = TcpListener::bind(&addr).await?;
loop {
    let (stream, _) = listener.accept().await?;
    let service = service.clone();
    // Spawn a new Tokio task for every incoming HTTP request
    tokio::spawn(handle_request(stream, service));
}

Inside each spawned task, the request is parsed, the appropriate routing algorithm is instantiated, and its run_stream is driven to completion. Since every driver channel is independent per request, different routing algorithms (random, classifier, stage router, etc.) execute concurrently without interference.

Model-Call Concurrency and Observability

The actual LLM calls are performed by the libsy-llm-client crate. Its run function spawns a Tokio task that sends requests to upstream providers and streams responses back.

When the driver's call_model method executes, it:

1. Records observability data via observability::record_llm_call in [crates/libsy/src/observability.rs](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy/src/observability.rs)
2. Captures latency, token usage, and outcome
3. Returns the response to the algorithm for next-step decisions

This design ensures that concurrent requests remain observable and debuggable, with per-request metrics isolated from other traffic.

Back-Pressure and Failure Handling

Switchyard's concurrency model includes robust mechanisms for handling load and failures:

Channel-based back-pressure – The MPSC channel's size of 1 ensures the algorithm cannot outrun the consumer, preventing memory exhaustion under high load.

Graceful degradation – If the consumer drops the response channel, the algorithm receives a DriverError::ResponseDropped and aborts cleanly rather than hanging.

Error propagation – Model provider errors return as Result<Response> values, allowing each algorithm to implement its own retry, fall-through, or abort logic.

Practical Usage: Concurrent Requests in Python

End-users interact with Switchyard through Python launchers that transparently leverage the Rust concurrency model. Here's how to run concurrent requests with different routing algorithms:

import asyncio
from switchyard.cli.launchers.codex_cli_launcher import CodexCLILauncher

async def main():
    # Create launcher; routing algorithm selected via --routing-algorithm flag

    launcher = CodexCLILauncher(model="tb-lite-random-routing")
    async with launcher:
        # Switchyard runs this routing algorithm concurrently with others

        response = await launcher.run_chat(
            messages=[{"role": "user", "content": "Hello"}]
        )
        print(response)

asyncio.run(main())

For explicit comparison of multiple algorithms:

import asyncio
from switchyard.cli.launchers.claude_code_launcher import ClaudeCodeLauncher

async def request(launcher, prompt):
    async with launcher:
        return await launcher.run_chat(
            messages=[{"role": "user", "content": prompt}]
        )

async def main():
    # Two launchers using different routing strategies

    rand = ClaudeCodeLauncher(model="tb-lite-random-routing")
    stage = ClaudeCodeLauncher(model="stage-router")

    results = await asyncio.gather(
        request(rand, "Tell me a joke."),
        request(stage, "Summarize this article...")
    )
    for r in results:
        print(r)

asyncio.run(main())

Low-Level Driver Access in Rust

For custom algorithm development, the Driver API provides direct control over concurrent execution:

use switchyard_libsy::Driver;
use switchyard_protocol::{Decision, Request, Response};

pub async fn my_algorithm(driver: Driver) -> Result<Response, Box<dyn Error>> {
    // First model call
    let req = Request::default();
    let resp = driver.call_model(req, vec!["model-a".into()], true).await?;

    // Record routing decision
    let decision = Decision::new("model-a".into(), "my_algorithm".into());
    driver.decide(decision).await?;

    // Complete the request
    driver.finish(Ok(resp.clone())).await?;
    Ok(resp)
}

Summary

Switchyard's concurrent request management rests on four key architectural decisions:

• Per-request isolation via dedicated Driver instances with private MPSC channels
• Natural back-pressure through bounded channels with capacity 1
• Tokio task-per-request scheduling in the Axum-based server
• Async step streaming that allows multiple in-flight model calls per request

This design enables any number of routing algorithms to process requests in parallel while maintaining complete state independence and observable behavior.

Frequently Asked Questions

What prevents different routing algorithms from interfering with each other?

Each request receives its own Driver with an independent MPSC channel. No shared state exists between drivers, so algorithms execute in complete isolation. The server spawns a separate Tokio task for every HTTP request, and task boundaries enforce this separation.

How does Switchyard handle a surge of concurrent requests?

The bounded channel (capacity 1) between each algorithm and its consumer provides natural back-pressure. Algorithms cannot emit new steps until previous ones complete, preventing unbounded queue growth. Tokio's work-stealing scheduler distributes tasks across runtime threads for efficient CPU utilization.

Can a single request make multiple model calls concurrently?

Yes. Because Step items are awaited asynchronously, algorithms can issue multiple call_model operations before awaiting their results. The stage router exploits this for fallback scenarios, though most algorithms serialize calls based on routing logic requirements.

What happens if a model provider is slow or fails?

Slow responses apply back-pressure through the channel. Failures propagate as Result<Response> errors to the algorithm, which decides on retry, alternative routing, or abort. The driver records all outcomes via observability::record_llm_call for debugging and metrics.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}