How to Implement Custom Routing Algorithms in the libsy Crate: A Complete Guide

Implement a custom routing algorithm in the libsy crate by implementing the Algorithm trait in crates/libsy/src/core/algorithm.rs, using the Driver to emit decisions and offload model calls.

The libsy crate provides the core routing engine for NVIDIA's Switchyard framework. Whether you're building affinity-based routing, model hedging, or custom heuristics, understanding how to implement custom routing algorithms requires deep familiarity with the Algorithm trait and its asynchronous Driver interface.

Understanding the Algorithm Trait

The foundation of custom routing is the Algorithm trait defined in crates/libsy/src/core/algorithm.rs. This trait represents an asynchronous, Arc-compatible object that transforms a Request into a Response through interaction with a host-provided Driver.

Required Methods

Your implementation must provide two core methods:

• fn name(&self) -> &str — Returns a low-cardinality identifier used for tracing and observability.
• async fn route(self: Arc<Self>, driver: Driver, request: Request) -> Result<Response> — Contains your complete routing logic.

The trait's default run_stream implementation (located at line 76 in algorithm.rs) wraps your route method with panic-catching and task spawning, guaranteeing a terminal Step::Done or Step::Error even if your algorithm panics.

Thread Safety Requirements

Your Algorithm implementation must be thread-safe (Send + Sync). A single Arc-wrapped instance can be shared across many concurrent requests, making proper synchronization essential for any shared mutable state.

The Driver Interface: Your Path to Host Interaction

The Driver struct (defined at line 91 in algorithm.rs) serves as your algorithm's only channel to the outside world. Each request receives a fresh Driver via Driver::new (line 100), ensuring isolation of step channels.

Key Driver Methods

Method	Signature	Purpose
call_model	async fn call_model(&self, request: Request, models: Vec<ModelId>, is_answer_call: bool) -> Result<Response>	Publish a model call and await the host's fulfillment
decide	async fn decide(&self, decision: Decision) -> Result<()>	Emit a routing decision recorded in the trace

Use driver.decide to publish intermediate routing choices for observability. Use driver.call_model to offload actual model inference, setting is_answer_call to true for the final answer-generating call.

Building Your First Custom Algorithm

Follow these concrete steps to implement custom routing algorithms in libsy.

Step 1: Create Your Algorithm Struct

Define a struct holding any static configuration or shared state:

// File: crates/libsy/src/algorithms/first_target.rs
use async_trait::async_trait;
use std::sync::Arc;
use switchyard_protocol::{Decision, ModelId, Request, Response};
use crate::{driver::Driver, Algorithm, Result};

pub struct FirstTarget {
    targets: Vec<ModelId>,
}

Step 2: Implement the Algorithm Trait

Provide the required trait implementation:

impl FirstTarget {
    pub fn new(targets: Vec<ModelId>) -> Self {
        Self { targets }
    }
}

#[async_trait]
impl Algorithm for FirstTarget {
    fn name(&self) -> &str {
        "first_target"
    }

    async fn route(
        self: Arc<Self>,
        driver: Driver,
        request: Request,
    ) -> Result<Response> {
        let first = self
            .targets
            .first()
            .ok_or(crate::LibsyError::NoTargets)?
            .clone();

        let decision = Decision::new(first.clone(), true);
        driver.decide(decision).await?;

        driver.call_model(request, vec![first], true).await
    }
}

Step 3: Register and Expose (Optional)

To expose your algorithm via the public API, add a factory function in crates/libsy/src/lib.rs:

// In crates/libsy/src/lib.rs
pub mod algorithms;
pub use algorithms::first_target::FirstTarget;

Advanced Patterns: Concurrency and Hedging

The libsy crate supports sophisticated routing patterns through its concurrency model. The driver tracks parallel calls via a FuturesUnordered pool (see the drive function at line 31 in algorithm.rs).

Implementing Model Hedging

Spawn multiple model calls concurrently and await the first successful result:

use futures::stream::{FuturesUnordered, StreamExt};

async fn route(
    self: Arc<Self>,
    driver: Driver,
    request: Request,
) -> Result<Response> {
    let mut futures = FuturesUnordered::new();
    
    for model in &self.targets {
        let driver_clone = driver.clone();
        let req = request.clone();
        let model = model.clone();
        
        futures.push(async move {
            driver_clone.call_model(req, vec![model], true).await
        });
    }
    
    // Return first successful response
    while let Some(result) = futures.next().await {
        if let Ok(response) = result {
            return Ok(response);
        }
    }
    
    Err(crate::LibsyError::AllTargetsFailed)
}

Protecting Shared State

Use Arc<Mutex<_>> or parking_lot::RwLock for mutable state shared across requests:

use parking_lot::RwLock;
use std::collections::HashMap;

pub struct CachedAlgorithm {
    cache: Arc<RwLock<HashMap<String, ModelId>>>,
}

Testing Your Custom Algorithm

The libsy crate provides robust testing utilities in crates/libsy/src/core/testing.rs.

Using the test_drive Helper

// File: crates/libsy/src/algorithms/first_target_tests.rs
#[cfg(test)]
mod tests {
    use super::*;
    use crate::core::testing::{test_drive, echo};
    use switchyard_protocol::{text_request, ModelId};

    fn make_algo() -> Arc<dyn Algorithm> {
        Arc::new(FirstTarget::new(vec![ModelId::from("demo-model")]))
    }

    #[tokio::test]
    async fn selects_first_target() -> Result<()> {
        let algo = make_algo();
        let request = Request {
            llm_request: text_request(
                Some("auto".into()), 
                "hello".into()
            ),
            raw_request: None,
            metadata: None,
        };
        
        let (trace, response) = test_drive(algo, request, echo()).await?;
        
        assert_eq!(trace[0].selected_model_id(), "demo-model");
        assert_eq!(
            response.llm_response.as_agg().unwrap().completion_text(),
            "demo-model"
        );
        Ok(())
    }
}

The echo helper returns the model name as the response text, making verification straightforward.

Key Source Files Reference

File	Purpose
crates/libsy/src/core/algorithm.rs	Algorithm trait, Driver, CallModel, Step, drive loop
crates/libsy/src/algorithms/passthrough.rs	Minimal forwarding reference implementation
crates/libsy/src/algorithms/rand.rs	Random target selection with state handling
crates/libsy/src/algorithms/util/target_selector.rs	Reusable target selection utilities
crates/libsy/src/core/testing.rs	test_drive, echo, and test harness helpers
crates/libsy/src/lib.rs	Public API re-exports

Summary

• Implement the Algorithm trait in crates/libsy/src/core/algorithm.rs to create custom routing logic.
• Use the Driver interface to emit decisions via decide() and offload model calls via call_model().
• Ensure thread safety (Send + Sync) since algorithms run under Arc across concurrent requests.
• Leverage built-in testing with test_drive and echo from crates/libsy/src/core/testing.rs.
• Study reference implementations in passthrough.rs and rand.rs for patterns on state management and randomization.
• Add custom algorithms without Python changes—libsy is deliberately minimal with all routing logic in Rust.

Frequently Asked Questions

What is the minimum code needed to implement a custom routing algorithm?

Define a struct implementing the Algorithm trait with name() and async fn route(). The route method receives a Driver and Request, uses driver.decide() to record routing choices, and returns the result of driver.call_model(). See crates/libsy/src/algorithms/passthrough.rs for the minimal working example.

How does the Driver handle concurrent model calls?

The Driver tracks pending calls in a FuturesUnordered pool via the internal drive function at line 31 of algorithm.rs. You can spawn multiple call_model futures concurrently; the driver manages their lifecycle and emits corresponding Step items to the host.

Can I share state between different requests to the same algorithm?

Yes, but you must protect shared mutable state with synchronization primitives like Arc<Mutex<_>> or parking_lot::RwLock. The Algorithm trait requires Send + Sync, and each request holds an Arc clone of your algorithm instance.

How do I test my custom algorithm without a full Switchyard deployment?

Use the test_drive helper from crates/libsy/src/core/testing.rs. This function runs your algorithm against a mock host, returning the complete trace and final response. The echo helper simulates a model that returns its own ID, enabling verification of routing decisions.

Where should I place my custom algorithm files?

Create new files under crates/libsy/src/algorithms/ following the existing naming convention (my_algo.rs for implementation, my_algo_tests.rs for tests). Reference implementations in passthrough.rs and rand.rs demonstrate the expected structure and patterns.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}