How to Configure Custom Routing Algorithms in a Switchyard TOML Deployment File

Configure custom routing algorithms in Switchyard by implementing the Algorithm trait in Rust, registering it in the AlgorithmRegistry, and referencing the algorithm name in your TOML deployment file under the algorithm key in a route definition.

Switchyard, NVIDIA's open-source LLM routing framework, uses TOML deployment files to define request routing topologies at server startup. The configuration system parses these files through the ServerConfig struct in crates/switchyard-server/src/config.rs, where each route can specify a custom routing algorithm that controls how requests are dispatched, retried, or combined across model providers.

Understanding the TOML Configuration Structure

The TOML deployment file consists of a [global] section for server-wide settings and one or more [[routes]] arrays that define processing pipelines. According to the Switchyard source code, route configuration is handled by the RouteConfig struct, which optionally contains an algorithm field pointing to an AlgorithmConfig.

In crates/switchyard-server/src/config.rs, the toml::from_str deserialization populates these structures, and the server_state_from_toml function wires algorithms into the request-processing pipeline. A minimal route definition with a custom algorithm looks like this:

[[routes]]
name = "custom_route"
model = { provider = "openai", model = "gpt-4o-mini" }
algorithm = "my_custom_algorithm"

Step 1: Implement the Algorithm Trait

Custom algorithms must implement the Algorithm trait defined in the libsy crate. Built-in implementations in crates/libsy/src/algorithms/—including passthrough, rand, and stage—serve as reference examples.

Create a new file in crates/libsy/src/algorithms/ with your implementation:

// crates/libsy/src/algorithms/my_stage.rs
use crate::algorithms::Algorithm;
use crate::core::processor::ProcessorContext;
use crate::{Request, Response, SwitchyardError};

#[derive(Debug, Default)]
pub struct MyStage;

impl Algorithm for MyStage {
    fn name(&self) -> &'static str {
        "my_stage"
    }

    async fn route(
        &self,
        ctx: ProcessorContext,
        request: Request,
    ) -> Result<Response, SwitchyardError> {
        // Custom routing logic: inspect, modify, retry, or fallback
        self.next_route(ctx, request).await
    }
}

The name() method returns the identifier used in TOML files. The route() method receives the processor context and request, enabling complex decision-making based on request content, latency budgets, or model availability.

Step 2: Register the Algorithm

Algorithms must be registered in the AlgorithmRegistry before the server can resolve them by name. Registration typically occurs in crates/libsy/src/lib.rs:

// crates/libsy/src/lib.rs
mod algorithms {
    pub mod my_stage;
    // ...other algorithm modules
}

use algorithms::my_stage::MyStage;

pub fn register_algorithms(registry: &mut AlgorithmRegistry) {
    registry.register(Box::new(MyStage::default()));
    // ...register other algorithms
}

The registry maps string names to algorithm instances, enabling dynamic lookup during TOML parsing in server_state_from_toml.

Step 3: Reference in TOML Deployment

With implementation and registration complete, reference the algorithm by its registered name in any route. Full examples are available in benchmark/server-configs/:

# benchmark/server-configs/custom-deployment.toml

[global]
port = 4000
log_level = "info"

[[routes]]
name = "staging_route"
model = { provider = "openai", model = "gpt-4o-mini" }
algorithm = "my_stage"

Start the server with your deployment file:

switchyard-server --config benchmark/server-configs/custom-deployment.toml

Key Source Files for Custom Algorithm Development

File	Purpose
crates/switchyard-server/src/config.rs	ServerConfig, RouteConfig, AlgorithmConfig definitions; TOML parsing via toml::from_str
crates/libsy/src/algorithms/	Built-in algorithm implementations and the Algorithm trait
crates/libsy/src/lib.rs	AlgorithmRegistry and registration functions
cli/defaults/openrouter.toml	Default deployment configuration with example routes
benchmark/server-configs/*.toml	Production-ready TOML examples for testing and benchmarking

Common Algorithm Patterns

Based on the built-in implementations in crates/libsy/src/algorithms/, effective custom algorithms typically handle:

• Request inspection — parse prompts to route by content type or complexity
• Latency-based failover — escalate to faster models when deadlines approach
• Broadcast and combine — query multiple models and synthesize responses
• Budget enforcement — track token costs and downgrade when limits near

Summary

• Implement the Algorithm trait in crates/libsy/src/algorithms/ with a unique name() return value
• Register your algorithm in crates/libsy/src/lib.rs via AlgorithmRegistry::register()
• Reference the algorithm by name in TOML route definitions using algorithm = "<name>"
• Validate your deployment against examples in benchmark/server-configs/

Frequently Asked Questions

What happens if I specify an algorithm name that isn't registered?

The server will fail to start with a configuration error during server_state_from_toml parsing. The TOML deserializer validates algorithm names against the AlgorithmRegistry; unregistered names trigger an error before the server accepts any requests.

Can I use different algorithms for different routes in the same deployment?

Yes. Each [[routes]] entry in your TOML file can specify a different algorithm value. The RouteConfig struct supports per-route algorithm selection, allowing mixed deployments where some routes use built-in algorithms while others use custom implementations.

Do I need to recompile Switchyard to add a custom algorithm?

Yes. Custom algorithms are compiled Rust code that must be linked into the server binary. The TOML file references the algorithm by name, but the implementation must exist in the compiled libsy crate. For runtime-pluggable algorithms, you would need to implement a dynamic loading mechanism separate from the native registry system.

Where can I find complete working examples of custom algorithms?

The benchmark/server-configs/ directory contains full TOML deployments, and crates/libsy/src/algorithms/stage.rs demonstrates a non-trivial algorithm implementation. The passthrough algorithm in the same directory provides the simplest possible reference implementation for new developers.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}