How to Embed Switchyard's Routing Algorithms in a Rust Application Using switchyard-libsy

switchyard-libsy is the core Rust crate that exposes provider-neutral routing algorithms as a stream of Step values, letting you embed LLM traffic orchestration into any existing transport or proxy layer without pulling in HTTP dependencies.

Embedding Switchyard's routing algorithms in your Rust application gives you fine-grained control over how LLM requests are distributed across model targets. The switchyard-libsy crate, located in the NVIDIA-NeMo/Switchyard repository, implements this logic as a pure algorithmic layer that yields routing decisions rather than performing network calls itself.

Architecture of switchyard-libsy

The crate follows a strict separation between routing logic and transport execution. This design lets you plug Switchyard into proxies, gateways, or custom orchestrators without modifying your existing HTTP stack.

Component	Responsibility	Key Types
switchyard-libsy	Core routing algorithms (Passthrough, Random, StageRouter)	Algorithm, Step, StepStream
switchyard-protocol	Provider-neutral request/response contracts	Request, Response, Decision
switchyard-llm-client	Ready-made consumer that drives StepStream and performs HTTP calls	run function
Host Application	Supplies model targets, executes HTTP calls, processes results	—

The central abstraction is the Algorithm trait. When you invoke Algorithm::run_stream, you receive a StepStream that yields Step values. Your application interprets each step—typically calling a model when asked—and feeds the resulting Response back via Algorithm::drive. The algorithm eventually produces a Decision recording which target was chosen and why.

Setting Up switchyard-libsy

Add the crates to your Cargo.toml:

[dependencies]
async-trait = "0.1"
futures = "0.3"
switchyard-libsy = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" }
switchyard-protocol = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" }
tokio = { version = "1", features = ["macros", "rt"] }

See crates/libsy/README.md in the repository for the complete dependency list and version compatibility notes.

Embedding the Passthrough Algorithm

The Passthrough algorithm in crates/libsy/src/algorithms/passthrough.rs is the simplest routing strategy—it unconditionally forwards all requests to a single model target. This makes it ideal for learning the embedding pattern or for deployments with dedicated model instances.

use switchyard_libsy::{Algorithm, Passthrough};
use switchyard_protocol::{Request, Response};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // 1. Instantiate the algorithm with a fixed target.
    let algo = Passthrough::new("my-model".into());

    // 2. Build a provider-neutral request.
    let request = Request::new_chat(
        "You are a helpful assistant.".into(),
        vec!["Explain quantum entanglement.".into()],
    );

    // 3. Obtain the step stream from the algorithm.
    let mut stream = algo.run_stream(request);

    // 4. Drive the stream: respond to each step the algorithm emits.
    while let Some(step) = stream.next().await {
        match step {
            switchyard_libsy::Step::CallModel { candidates, .. } => {
                // Passthrough guarantees exactly one candidate.
                let target = &candidates[0];
                
                // Your HTTP call goes here—switchyard-libsy has no opinion on transport.
                let response: Response = call_llm(target, step.request).await?;
                
                // Feed the response back to advance the algorithm.
                stream.drive(response).await?;
            }
            switchyard_libsy::Step::Done { decision, .. } => {
                println!("Routed to: {}", decision.target);
                break;
            }
            _ => {}
        }
    }
    Ok(())
}

async fn call_llm(target: &str, request: Request) -> anyhow::Result<Response> {
    // Your existing HTTP client logic.
    todo!()
}

Key types are re-exported from crates/libsy/src/lib.rs: Algorithm, Step, and StepStream form the public API surface.

Using StageRouter for Signal-Driven Routing

The StageRouter algorithm in crates/libsy/src/algorithms/stage.rs implements more sophisticated routing based on external signals—tool usage, progress indicators, or custom application events. This powers use cases like coding-agent launchers that hand off between specialized models.

use switchyard_libsy::{
    Algorithm, StageRouter, StageRouterConfig, ToolSignals,
};
use switchyard_protocol::Request;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Configure primary and fallback targets.
    let config = StageRouterConfig {
        primary_target: "coding-agent".into(),
        fallback_target: Some("judge-model".into()),
        ..Default::default()
    };
    let router = StageRouter::new(config);

    // Signals drive routing decisions (e.g., tool execution progress).
    let signals = ToolSignals::default();

    let request = Request::new_chat(
        "Refactor this function to use async/await".into(),
        vec![],
    );

    let mut stream = router.run_stream(request);
    
    while let Some(step) = stream.next().await {
        match step {
            switchyard_libsy::Step::CallModel { candidates, context } => {
                // StageRouter may rank candidates by signal state.
                let target = select_best(candidates, &signals);
                let response = call_llm(target, context.request).await?;
                stream.drive(response).await?;
            }
            switchyard_libsy::Step::Done { decision, .. } => {
                println!("Final routing decision: {:?}", decision);
                break;
            }
            _ => {}
        }
    }
    Ok(())
}

Signal utilities live in crates/libsy/src/algorithms/util/tool_signals.rs. The ToolSignals type lets you feed runtime information into the router without coupling your signal sources to the algorithm implementation.

Driving Routing with the Built-In Client

If you prefer not to implement the StepStream driver yourself, the switchyard-llm-client crate provides a drop-in solution. The run function in crates/libsy-llm-client/src/run.rs consumes a StepStream and performs HTTP calls using your configured client.

use switchyard_libsy_llm_client::run;
use switchyard_libsy::{Algorithm, Passthrough};
use switchyard_protocol::Request;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let algo = Passthrough::new("my-model".into());
    let request = Request::new_chat("Tell me a joke".into(), vec![]);

    // `run` handles the stream loop and returns the final Decision.
    let decision = run(algo, request, /* your http_client */).await?;
    println!("Model {} responded successfully", decision.target);
    
    Ok(())
}

This approach sacrifices some control for convenience—you lose the ability to interpose custom logic between steps, but gain a working integration in minimal code.

Key Integration Points

When embedding switchyard-libsy in production applications, focus on these implementation locations:

• crates/libsy/src/lib.rs — Public API re-exports; start here to understand available algorithms and types.
• crates/libsy/src/algorithms/passthrough.rs — Reference implementation for single-target routing.
• crates/libsy/src/algorithms/stage.rs — Complex signal-driven routing with fallback support.
• crates/protocol/src/lib.rs — Request, Response, and Decision structures for provider-neutral communication.
• crates/libsy-llm-client/src/run.rs — Production-ready driver if you don't need custom step handling.

The algorithm-stream pattern makes switchyard-libsy embeddable in any async Rust runtime. Because networking is entirely your responsibility, you retain full control over connection pooling, retry policies, observability hooks, and authentication.

Summary

• switchyard-libsy exposes routing algorithms as StepStream producers, not HTTP clients.
• Embed algorithms by instantiating Passthrough, StageRouter, or custom Algorithm implementations, then driving their StepStream with your transport code.
• Three integration depths: manual stream driving (full control), ToolSignals for dynamic routing, or switchyard-llm-client::run for convenience.
• Zero HTTP dependencies in the core crate—bring your own reqwest, hyper, or internal client.
• Provider-neutral protocol via switchyard-protocol keeps your application decoupled from specific LLM API shapes.

Frequently Asked Questions

Does switchyard-libsy make HTTP requests automatically?

No. The crate is intentionally transport-agnostic. It yields Step::CallModel values that your application interprets and executes. This design lets you integrate with existing proxy infrastructure, custom connection pools, or specialized authentication layers without modification. The switchyard-llm-client crate provides an optional HTTP implementation if you need one.

How do I choose between Passthrough and StageRouter?

Passthrough in crates/libsy/src/algorithms/passthrough.rs routes all traffic to a single fixed target—use it for dedicated model deployments or when learning the embedding pattern. StageRouter in crates/libsy/src/algorithms/stage.rs responds to runtime signals and supports fallback targets—use it for dynamic scenarios like coding agents that may need to hand off between specialized models based on tool usage or progress indicators.

Can I implement custom routing algorithms?

Yes. Implement the Algorithm trait from crates/libsy/src/lib.rs. Your implementation must provide run_stream, which returns a StepStream yielding your custom Step sequence. The trait design ensures your algorithm remains decoupled from transport concerns while interoperating with existing Switchyard tooling.

What information does a Decision contain?

The Decision type from switchyard-protocol records the final target model ID, the routing path taken (which algorithms or stages processed the request), and metadata about why that choice was made. Inspect Decision after your StepStream yields Step::Done to log routing telemetry or enforce post-hoc validation policies.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}