# Project Structure of the NVIDIA Switchyard Repository: Rust Core, Python Bindings, and Proxy Architecture

> Explore the NVIDIA Switchyard repository structure. Discover its Rust core for routing, Python bindings via PyO3, and proxy architecture. Learn how this mixed codebase powers efficient network solutions.

- Repository: [NVIDIA-NeMo/Switchyard](https://github.com/NVIDIA-NeMo/Switchyard)
- Tags: architecture
- Published: 2026-08-21

---

**The Switchyard repository is organized as a mixed Rust-Python codebase with a Rust workspace containing six specialized crates for routing algorithms and proxy serving, a Python package providing PyO3 bindings and wrappers, and dedicated directories for documentation, benchmarks, and development tooling.**

The NVIDIA-NeMo/Switchyard repository implements a high-performance LLM routing proxy as a polyglot codebase designed for both standalone deployment and library embedding. Understanding the project structure is essential for developers looking to extend routing algorithms, integrate the proxy server, or leverage the core Rust libraries from Python. The codebase deliberately separates the native Rust implementation from Python integration layers while maintaining tight coupling through automated bindings generated via PyO3.

## Top-Level Directory Layout

The repository root organizes code by functional concern rather than language, with seven primary directories handling distinct responsibilities from documentation to benchmarking.

### The Rust Workspace (switchyard_rust/)

The `switchyard_rust/` directory contains the complete Rust workspace defined by the [`Cargo.toml`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/Cargo.toml) at the repository root. This workspace aggregates six crates under the `crates/` subdirectory:

- **`libsy/`** - Core routing algorithms including Random, StageRouter, and LLM-based classifiers
- **`protocol/`** - Provider-neutral request/response types shared across the system
- **`switchyard-server/`** - Standalone HTTP proxy executable
- **`switchyard-translation/`** - OpenAI/Anthropic format conversion layer
- **`switchyard-llm-client/`** - HTTP client for backend model servers
- **`switchyard-py/`** - PyO3 bindings that compile into the Python wheel

### The Python Package (switchyard/)

The `switchyard/` directory provides the importable Python package (`import switchyard`). Key contents include:

- **[`__init__.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/__init__.py)** - Package version definition and entry point
- **`libsy/`** - Typed Python wrappers that expose Rust algorithms with ergonomic Pythonic interfaces

### Documentation and Examples

The `docs/` directory contains **MkDocs**-rendered user guides covering routing algorithms and internal architecture, located at paths like `docs/routing_algorithms/` and [`docs/architecture.md`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/docs/architecture.md). The `examples/` directory provides runnable snippets including [`libsy.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/libsy.py) for embedded algorithm usage and `prometheus/` for metrics configuration.

### Testing and Development Infrastructure

The `tests/` directory houses the **Pytest** suite for Python-side validation, including [`test_libsy_minimal_bindings.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/test_libsy_minimal_bindings.py) which verifies Rust-Python integration. The `scripts/` directory contains development utilities like [`benchmark_routing_algorithms.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/benchmark_routing_algorithms.py) and [`run_local_soak_test.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/run_local_soak_test.py), while `benchmark/` holds Dockerfiles and data for performance testing.

## Rust Crate Architecture

The Rust implementation follows a modular crate design that enforces clean separation between routing logic, protocol handling, and serving infrastructure.

### Core Routing Algorithms (crates/libsy/)

Located at `switchyard_rust/crates/libsy/`, this crate implements the routing decision logic. The [`README.md`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/README.md) in this directory documents the public API for algorithms. These algorithms determine backend selection based on configured strategies, with implementations using async streams via `run_stream()` methods.

### Protocol and Translation Layers

The `crates/protocol/` directory defines provider-neutral request/response structs used across the system. The `crates/switchyard-translation/` crate handles conversion between OpenAI/Anthropic formats and backend-native formats, enabling Switchyard to present a unified API while communicating with diverse model servers like vLLM, NVIDIA NIM, or Ollama.

### Server Implementation (crates/switchyard-server/)

This crate produces the standalone binary installable via `cargo install switchyard-server`. The server coordinates routing decisions, translation, and backend communication. Configuration occurs through TOML files validated via the `--dry-run` flag before startup.

### Python Bindings (crates/switchyard-py/)

The `switchyard-py` crate uses **PyO3** to generate Python-compatible extension modules. This crate bridges the Rust workspace with the `switchyard/` Python package, compiling Rust structs and functions into `.so` or `.pyd` files that Python imports seamlessly.

## Python Integration Layer

The Python side prioritizes ergonomic access to Rust performance without exposing implementation complexity.

### Package Initialization

The root [`switchyard/__init__.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard/__init__.py) handles package versioning and imports the compiled extension from `switchyard_rust/crates/switchyard-py`. Users interact with high-level classes like `SwitchyardClient` while actual computation occurs in optimized Rust code.

### Algorithm Wrappers

The [`switchyard/libsy/algorithms.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard/libsy/algorithms.py) file exposes classes like `Random` and `StageRouter` that wrap their Rust counterparts. These typed wrappers provide IDE-friendly type hints and handle conversion between Python dictionaries and Rust structs.

## Building and Running the System

Developers interact with the repository through multiple entry points depending on deployment needs.

### Running the Standalone Server

For proxy deployment, build and run the Rust server:

```bash

# Install from within the repository

cargo install --locked --path switchyard_rust/crates/switchyard-server

# Validate configuration before starting

switchyard-server --config routes.toml --dry-run

# Start the proxy on port 4000

switchyard-server --config routes.toml --host 127.0.0.1 --port 4000

```

### Embedding in Python Applications

For programmatic routing without the standalone server:

```python
from switchyard.libsy import Random
from switchyard import SwitchyardClient

# Initialize routing algorithm

algo = Random(targets=["gpt-4o", "llama-2-70b"])

# Create client with algorithm

client = SwitchyardClient(algorithm=algo)

# Send request

response = client.chat(messages=[{"role": "user", "content": "Hello"}])

```

### Direct Rust Usage

Import the crates directly in Rust projects:

```toml
[dependencies]
switchyard-libsy = { git = "https://github.com/NVIDIA-NeMo/Switchyard", tag = "v0.2.0" }
switchyard-protocol = { git = "https://github.com/NVIDIA-NeMo/Switchyard", tag = "v0.2.0" }
tokio = { version = "1", features = ["macros", "rt"] }

```

Then implement routing:

```rust
use switchyard_libsy::{Random, Algorithm};

#[tokio::main]
async fn main() {
    let algo = Random::new(vec!["gpt-4o".into(), "llama-2".into()]);
    // Use algo.run_stream() for async routing decisions
}

```

## Summary

- The **Switchyard repository** separates concerns by placing Rust source in `switchyard_rust/` and Python integration in `switchyard/`
- The **Rust workspace** contains six crates: `libsy` (routing), `protocol` (types), `switchyard-server` (proxy), `switchyard-translation` (format conversion), `switchyard-llm-client` (HTTP), and `switchyard-py` (bindings)
- **Python users** import from the `switchyard` package, which loads PyO3-generated extensions built from `crates/switchyard-py/`
- **Configuration** happens via TOML files validated by the Rust server binary using the `--dry-run` flag
- **Testing** spans both languages with Pytest in `tests/` and Cargo tests in the respective crate directories

## Frequently Asked Questions

### What is the relationship between the switchyard and switchyard_rust directories?

The `switchyard/` directory contains the importable Python package that users install via pip, while `switchyard_rust/` contains the workspace of Rust crates that provide the actual implementation. The `crates/switchyard-py/` crate bridges these worlds by compiling Rust code into a Python extension module that [`switchyard/__init__.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard/__init__.py) loads and exposes to end users.

### How do I add a new routing algorithm to the codebase?

Implement the algorithm in `switchyard_rust/crates/libsy/src/`, following the trait definitions documented in [`crates/libsy/README.md`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy/README.md). After adding the Rust implementation, update [`switchyard/libsy/algorithms.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard/libsy/algorithms.py) to expose the new algorithm to Python users. Add integration tests in [`tests/test_libsy_minimal_bindings.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/tests/test_libsy_minimal_bindings.py) and unit tests in the Rust crate's `tests/` directory to ensure correctness across both languages.

### Where is the configuration schema defined for the standalone server?

The server accepts TOML configuration files validated against schemas documented in [`docs/reference/toml_schema.md`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/docs/reference/toml_schema.md). The Rust parsing logic resides in `crates/switchyard-server/`, which uses the `--dry-run` flag to validate configurations without starting the proxy. Example configurations appear in [`dev-server/config.toml`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/dev-server/config.toml) for reference during development.

### Can I use the routing algorithms without the full proxy server?

Yes. The `libsy` crate functions as a standalone library for embedding routing logic. Python users can import algorithms directly from `switchyard.libsy` without running the server binary, while Rust developers can depend only on `switchyard-libsy` and `switchyard-protocol` crates without pulling in server-specific dependencies like `switchyard-server` or `switchyard-translation`.