Project Structure of the NVIDIA Switchyard Repository: Rust Core, Python Bindings, and Proxy Architecture

The Switchyard repository is organized as a mixed Rust-Python codebase with a Rust workspace containing six specialized crates for routing algorithms and proxy serving, a Python package providing PyO3 bindings and wrappers, and dedicated directories for documentation, benchmarks, and development tooling.

The NVIDIA-NeMo/Switchyard repository implements a high-performance LLM routing proxy as a polyglot codebase designed for both standalone deployment and library embedding. Understanding the project structure is essential for developers looking to extend routing algorithms, integrate the proxy server, or leverage the core Rust libraries from Python. The codebase deliberately separates the native Rust implementation from Python integration layers while maintaining tight coupling through automated bindings generated via PyO3.

Top-Level Directory Layout

The repository root organizes code by functional concern rather than language, with seven primary directories handling distinct responsibilities from documentation to benchmarking.

The Rust Workspace (switchyard_rust/)

The switchyard_rust/ directory contains the complete Rust workspace defined by the Cargo.toml at the repository root. This workspace aggregates six crates under the crates/ subdirectory:

• libsy/ - Core routing algorithms including Random, StageRouter, and LLM-based classifiers
• protocol/ - Provider-neutral request/response types shared across the system
• switchyard-server/ - Standalone HTTP proxy executable
• switchyard-translation/ - OpenAI/Anthropic format conversion layer
• switchyard-llm-client/ - HTTP client for backend model servers
• switchyard-py/ - PyO3 bindings that compile into the Python wheel

The Python Package (switchyard/)

The switchyard/ directory provides the importable Python package (import switchyard). Key contents include:

• __init__.py - Package version definition and entry point
• libsy/ - Typed Python wrappers that expose Rust algorithms with ergonomic Pythonic interfaces

Documentation and Examples

The docs/ directory contains MkDocs-rendered user guides covering routing algorithms and internal architecture, located at paths like docs/routing_algorithms/ and docs/architecture.md. The examples/ directory provides runnable snippets including libsy.py for embedded algorithm usage and prometheus/ for metrics configuration.

Testing and Development Infrastructure

The tests/ directory houses the Pytest suite for Python-side validation, including test_libsy_minimal_bindings.py which verifies Rust-Python integration. The scripts/ directory contains development utilities like benchmark_routing_algorithms.py and run_local_soak_test.py, while benchmark/ holds Dockerfiles and data for performance testing.

Rust Crate Architecture

The Rust implementation follows a modular crate design that enforces clean separation between routing logic, protocol handling, and serving infrastructure.

Core Routing Algorithms (crates/libsy/)

Located at switchyard_rust/crates/libsy/, this crate implements the routing decision logic. The README.md in this directory documents the public API for algorithms. These algorithms determine backend selection based on configured strategies, with implementations using async streams via run_stream() methods.

Protocol and Translation Layers

The crates/protocol/ directory defines provider-neutral request/response structs used across the system. The crates/switchyard-translation/ crate handles conversion between OpenAI/Anthropic formats and backend-native formats, enabling Switchyard to present a unified API while communicating with diverse model servers like vLLM, NVIDIA NIM, or Ollama.

Server Implementation (crates/switchyard-server/)

This crate produces the standalone binary installable via cargo install switchyard-server. The server coordinates routing decisions, translation, and backend communication. Configuration occurs through TOML files validated via the --dry-run flag before startup.

Python Bindings (crates/switchyard-py/)

The switchyard-py crate uses PyO3 to generate Python-compatible extension modules. This crate bridges the Rust workspace with the switchyard/ Python package, compiling Rust structs and functions into .so or .pyd files that Python imports seamlessly.

Python Integration Layer

The Python side prioritizes ergonomic access to Rust performance without exposing implementation complexity.

Package Initialization

The root switchyard/__init__.py handles package versioning and imports the compiled extension from switchyard_rust/crates/switchyard-py. Users interact with high-level classes like SwitchyardClient while actual computation occurs in optimized Rust code.

Algorithm Wrappers

The switchyard/libsy/algorithms.py file exposes classes like Random and StageRouter that wrap their Rust counterparts. These typed wrappers provide IDE-friendly type hints and handle conversion between Python dictionaries and Rust structs.

Building and Running the System

Developers interact with the repository through multiple entry points depending on deployment needs.

Running the Standalone Server

For proxy deployment, build and run the Rust server:

# Install from within the repository

cargo install --locked --path switchyard_rust/crates/switchyard-server

# Validate configuration before starting

switchyard-server --config routes.toml --dry-run

# Start the proxy on port 4000

switchyard-server --config routes.toml --host 127.0.0.1 --port 4000

Embedding in Python Applications

For programmatic routing without the standalone server:

from switchyard.libsy import Random
from switchyard import SwitchyardClient

# Initialize routing algorithm

algo = Random(targets=["gpt-4o", "llama-2-70b"])

# Create client with algorithm

client = SwitchyardClient(algorithm=algo)

# Send request

response = client.chat(messages=[{"role": "user", "content": "Hello"}])

Direct Rust Usage

Import the crates directly in Rust projects:

[dependencies]
switchyard-libsy = { git = "https://github.com/NVIDIA-NeMo/Switchyard", tag = "v0.2.0" }
switchyard-protocol = { git = "https://github.com/NVIDIA-NeMo/Switchyard", tag = "v0.2.0" }
tokio = { version = "1", features = ["macros", "rt"] }

Then implement routing:

use switchyard_libsy::{Random, Algorithm};

#[tokio::main]
async fn main() {
    let algo = Random::new(vec!["gpt-4o".into(), "llama-2".into()]);
    // Use algo.run_stream() for async routing decisions
}

Summary

• The Switchyard repository separates concerns by placing Rust source in switchyard_rust/ and Python integration in switchyard/
• The Rust workspace contains six crates: libsy (routing), protocol (types), switchyard-server (proxy), switchyard-translation (format conversion), switchyard-llm-client (HTTP), and switchyard-py (bindings)
• Python users import from the switchyard package, which loads PyO3-generated extensions built from crates/switchyard-py/
• Configuration happens via TOML files validated by the Rust server binary using the --dry-run flag
• Testing spans both languages with Pytest in tests/ and Cargo tests in the respective crate directories

Frequently Asked Questions

What is the relationship between the switchyard and switchyard_rust directories?

The switchyard/ directory contains the importable Python package that users install via pip, while switchyard_rust/ contains the workspace of Rust crates that provide the actual implementation. The crates/switchyard-py/ crate bridges these worlds by compiling Rust code into a Python extension module that switchyard/__init__.py loads and exposes to end users.

How do I add a new routing algorithm to the codebase?

Implement the algorithm in switchyard_rust/crates/libsy/src/, following the trait definitions documented in crates/libsy/README.md. After adding the Rust implementation, update switchyard/libsy/algorithms.py to expose the new algorithm to Python users. Add integration tests in tests/test_libsy_minimal_bindings.py and unit tests in the Rust crate's tests/ directory to ensure correctness across both languages.

Where is the configuration schema defined for the standalone server?

The server accepts TOML configuration files validated against schemas documented in docs/reference/toml_schema.md. The Rust parsing logic resides in crates/switchyard-server/, which uses the --dry-run flag to validate configurations without starting the proxy. Example configurations appear in dev-server/config.toml for reference during development.

Can I use the routing algorithms without the full proxy server?

Yes. The libsy crate functions as a standalone library for embedding routing logic. Python users can import algorithms directly from switchyard.libsy without running the server binary, while Rust developers can depend only on switchyard-libsy and switchyard-protocol crates without pulling in server-specific dependencies like switchyard-server or switchyard-translation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}