How to Embed Switchyard Routing Algorithms in a Custom Rust Application

You can embed Switchyard routing algorithms in any Rust service by consuming the switchyard-libsy crate, which returns a StepStream that yields routing decisions (Step::CallModel) your application executes via its own HTTP stack, then feeds responses back through Algorithm::drive.

The NVIDIA-NeMo/Switchyard repository provides provider-neutral LLM traffic orchestration primitives through the switchyard-libsy crate. Because the core routing logic is completely decoupled from networking, you can embed sophisticated algorithms—such as Passthrough, Random, or StageRouter—into custom proxies, API gateways, or LLM orchestrators without importing heavy HTTP client dependencies.

Architecture of the Routing Library

The switchyard-libsy crate exposes a driver pattern that separates routing intelligence from transport execution. The central types defined in crates/libsy/src/lib.rs are:

• Algorithm – The routing engine trait (e.g., Passthrough, StageRouter).
• StepStream – An async stream produced by Algorithm::run_stream that yields routing steps.
• Step – Individual routing instructions, primarily Step::CallModel containing candidate target IDs.
• Decision – Final routing outcome exposed via switchyard_protocol::Decision.

Your host application acts as the driver: it consumes the StepStream, performs the actual HTTP calls to LLM endpoints, and pushes Response objects back into the algorithm via stream.drive. This design lets you integrate Switchyard into existing Rust async runtimes (Tokio, async-std) or custom proxy layers without architectural lock-in.

Adding switchyard-libsy to Your Project

Add the core crates to your Cargo.toml along with async utilities:

[dependencies]
async-trait = "0.1"
futures = "0.3"
tokio = { version = "1", features = ["macros", "rt"] }
switchyard-libsy = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" }
switchyard-protocol = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" }

The switchyard-protocol crate provides provider-neutral request/response contracts (Request, Response) used by all algorithms.

Driving Routing Algorithms with Custom Transport

Embedding Switchyard requires implementing a driver loop that bridges the StepStream to your HTTP stack. The pattern is consistent across all algorithms:

1. Instantiate an Algorithm (e.g., Passthrough::new).
2. Invoke algorithm.run_stream(request) to obtain a StepStream.
3. Poll the stream for Step::CallModel, execute the HTTP call to the candidate target, and capture the Response.
4. Drive the response back into the stream via stream.drive(response) to advance the state machine.
5. Collect the final Step::Done containing the Decision.

Basic Passthrough Routing

The Passthrough algorithm (implemented in crates/libsy/src/algorithms/passthrough.rs) demonstrates the minimal integration. It always routes to a single pre-configured target:

use switchyard_libsy::{Algorithm, Passthrough};
use switchyard_protocol::{Request, Response};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Initialize the algorithm with a static target
    let algo = Passthrough::new("my-model".into());
    
    // Build a provider-neutral request
    let request = Request::new_chat(
        "You are a helpful assistant.".into(),
        vec!["Explain quantum entanglement.".into()],
    );
    
    // Start the routing stream
    let mut stream = algo.run_stream(request);
    
    // Drive the stream
    while let Some(step) = stream.next().await {
        match step {
            switchyard_libsy::Step::CallModel { candidates, .. } => {
                // Host executes HTTP call to the first candidate
                let target = &candidates[0];
                let response: Response = your_http_client_call(target, &step.request).await?;
                
                // Feed response back to advance the algorithm
                stream.drive(response).await?;
            }
            switchyard_libsy::Step::Done { decision, .. } => {
                println!("Routed to: {}", decision.target);
                break;
            }
            _ => {}
        }
    }
    Ok(())
}

Signal-Driven Routing with StageRouter

For dynamic workloads, the StageRouter (defined in crates/libsy/src/algorithms/stage.rs) accepts external signals—such as tool usage or progress events—to inform routing decisions. You supply these signals via ToolSignals from crates/libsy/src/algorithms/util/tool_signals.rs:

use switchyard_libsy::{Algorithm, StageRouter, StageRouterConfig, ToolSignals};
use switchyard_protocol::Request;

// Configure primary and fallback targets
let config = StageRouterConfig {
    primary_target: "coding-agent".into(),
    fallback_target: Some("judge-model".into()),
    ..Default::default()
};

let router = StageRouter::new(config);

// Prepare signal stream (populated by your application logic)
let signals = ToolSignals::default();

// Build request and run stream
let request = Request::new_chat("Refactor this function".into(), vec![]);
let mut stream = router.run_stream(request);

// Drive loop identical to Passthrough, but routing decisions may change
// based on signals injected via router.update_signals()
while let Some(step) = stream.next().await {
    // Handle Step::CallModel and Step::Done as shown above
}

Using the Built-in LLM Client Driver

If you prefer not to implement the driver loop manually, the switchyard-llm-client crate provides a drop-in consumer. The run function in crates/libsy-llm-client/src/run.rs handles StepStream iteration and performs HTTP calls using your configured client:

use switchyard_libsy_llm_client::run;
use switchyard_libsy::{Algorithm, Passthrough};
use switchyard_protocol::Request;

let algo = Passthrough::new("my-model".into());
let request = Request::new_chat("Tell me a joke".into(), vec![]);

// `run` drives the stream and returns the final Decision
let decision = run(algo, request, /* your http_client */).await?;
println!("Selected model: {}", decision.target);

This approach trades flexibility for convenience, abstracting the Step::CallModel handling while preserving the algorithmic logic.

Summary

• switchyard-libsy provides decoupled routing algorithms that yield StepStream objects rather than making HTTP calls directly.
• Driver implementation requires polling Step::CallModel, executing the network request yourself, and feeding the Response back via stream.drive.
• Key source files include crates/libsy/src/lib.rs (public API), crates/libsy/src/algorithms/passthrough.rs (static routing), and crates/libsy/src/algorithms/stage.rs (signal-driven routing).
• switchyard-protocol defines provider-neutral Request and Response structures in crates/protocol/src/lib.rs.
• Optional convenience is available through switchyard-llm-client in crates/libsy-llm-client/src/run.rs for ready-made stream consumption.

Frequently Asked Questions

Does switchyard-libsy perform HTTP requests internally?

No. The crate is intentionally transport-agnostic. It yields Step::CallModel instructions that your host application must execute. This architecture allows embedding Switchyard into existing proxies or custom networking stacks without pulling in unwanted HTTP dependencies.

Which routing algorithms are available for embedding?

The library ships with several implementations in crates/libsy/src/algorithms/: Passthrough for static targets, Random for load distribution, and StageRouter for signal-driven orchestration. All implement the Algorithm trait defined in the crate root.

How do I feed custom telemetry into the StageRouter?

The StageRouter accepts ToolSignals (importable from crates/libsy/src/algorithms/util/tool_signals.rs). Your application updates this signal container with runtime telemetry—such as token counts, tool invocations, or latency metrics—and the router adjusts its Step generation accordingly when you call router.update_signals().

Can I use Switchyard routing without Tokio?

Yes. While the examples use Tokio, the Algorithm trait and StepStream rely only on standard futures::Stream and async-trait. You can drive the stream using any async runtime or even manually poll it in synchronous wrappers, provided you maintain the async drive contract for feeding responses back.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}