# Programming Languages Used in Switchyard: Rust Core, Python API, and More

> Discover the programming languages powering NVIDIA Switchyard. Learn how Rust drives the core engine and Python powers the API, optimizing your projects.

- Repository: [NVIDIA-NeMo/Switchyard](https://github.com/NVIDIA-NeMo/Switchyard)
- Tags: deep-dive
- Published: 2026-08-23

---

**TLDR: Switchyard uses Rust for its high-performance native server and routing engine, and Python for its public API, CLI tools, and orchestration scripts — connected via PyO3 bindings, with TOML and Bash handling configuration and CI.**

The NVIDIA-NeMo/Switchyard repository is a modern AI gateway that stands out for its deliberate **polyglot architecture**. Instead of limiting itself to a single language, Switchyard pairs a **Rust-based core server** with a **Python-facing client API**, giving developers both low-level performance and high-level ergonomics. As the source tree shows, every performance-critical component lives in the Rust `crates/` workspace, while the user-facing `switchyard` Python package orchestrates workflows and exposes the OpenAI-compatible interface. This article breaks down exactly which languages appear, where they live in the codebase, and why the split matters for developers.

## The Two Primary Languages in Switchyard

| Language | Role in Switchyard | Representative Source |
|----------|-------------------|-----------------------|
| **Rust** | Implements the high-performance native server, routing algorithms, protocol types, and the PyO3 bindings that expose the Rust core to Python. | [crates/switchyard-py/Cargo.toml](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-py/Cargo.toml) – defines the Rust crate that builds the Python bindings. |
| **Python** | Provides the public API (`switchyard` package), CLI utilities, integration scripts, and higher-level workflow orchestration. | [switchyard/__init__.py](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard/__init__.py) – entry point for the Python library. |

This composition is **by design**: as implemented in Switchyard, Rust supplies deterministic, low-latency processing while Python gives developers a familiar scripting interface and deep ecosystem integration.

### The Rust Core: Performance Where It Counts

The `crates/` directory at the repository root contains a workspace of Rust crates — `switchyard-libsy`, `switchyard-protocol`, `switchyard-server`, `switchyard-soak`, and others. These crates compile into two artifacts:

- A **native binary** (`switchyard-server`) that can run standalone, handling HTTP routing directly.
- A **shared library** (`_switchyard_rust`) that Python loads at runtime through **PyO3**.

On the Rust side, Switchyard handles high-throughput HTTP routing, protocol translation, metrics collection, and A/B testing. All of these operations run without the overhead of the Python interpreter, which keeps latency low even under heavy concurrent load.

### The Python Layer: Developer-Facing Convenience

The `switchyard` package offers a provider-neutral client API that mirrors the OpenAI and Anthropic SDKs. It exposes builder methods like `client.Chat` and `client.Completions`, and at runtime it loads the compiled Rust extension (`_switchyard_rust`) automatically. Helper scripts under `scripts/` and `examples/` are all written in Python, simplifying local testing, benchmark orchestration, and CI integration.

## How Rust and Python Work Together

The key glue is the **PyO3 binding** in [`crates/switchyard-py/Cargo.toml`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-py/Cargo.toml). This Rust crate compiles into a CPython extension module. When you install Switchyard from source or via a wheel, the build generates the native extension that the Python package imports. Effectively, you get:

1. A fast Rust skeleton that lazy-optimizes and offloads heavy traffic.
2. A comfortable Python surface that hides the low-level server complexity.

This mirrors common ecosystem patterns (like `pydantic-core` in Rust or ChromaDB), but in Switchyard it is baked into the routing and benchmark tooling rather than just data validation.

## Code Examples in Both Languages

### Python: Basic Switchyard Client

```python
import switchyard

# Create a synchronous client that talks to a locally-running Switchyard server

client = switchyard.client.SyncClient(base_url="http://localhost:4000")

# Send a simple chat request

response = client.chat(messages=[{"role": "user", "content": "Hello, Switchyard!"}])

print("Switchyard reply:", response["choices"][0]["message"]["content"])

```

The `switchyard.client` module lives in the Python package and forwards the request to the Rust server via the PyO3 binding.

### Rust: Minimal Server Entry Point (Excerpt)

```rust
// crates/switchyard-server/src/main.rs
use switchyard_server::Server;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Bind to the address supplied via CLI arguments
    let server = Server::bind("0.0.0.0:4000").await?;
    // Start handling incoming LLM traffic
    server.serve().await?;
    Ok(())
}

```

The server is compiled from the Rust crate `switchyard-server` and runs as a native binary, delivering the routing logic used by the Python client.

## Supporting Languages and Configuration Files

While Rust and Python are the two programming languages, Switchyard also relies on two auxiliary file formats for build and runtime configuration:

1. **TOML** – Used for manifest files ([`Cargo.toml`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/Cargo.toml), [`pyproject.toml`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/pyproject.toml)) and for custom route descriptors like [`benchmark/server-configs/tb-lite-single-opus-4-7.toml`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/benchmark/server-configs/tb-lite-single-opus-4-7.toml). These define which model providers, latency targets, and traffic splits the Rust server should honor.

2. **Bash / Shell** – Scripts such as [`benchmark/run-baseline.sh`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/benchmark/run-baseline.sh) orchestrate micro-benchmark runs, A/B pipelines, and CI triggering.

These aren't programming languages in the traditional sense, but they are essential support files that ensure the whole system builds and runs as one unit.

## Where to Find Them: Key Files

| File | Language | Why it matters |
|------|----------|----------------|
| [`crates/switchyard-py/Cargo.toml`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-py/Cargo.toml) | Rust (manifest) | Shows the Rust crate that builds the Python bindings (`pyo3` dependency). |
| [`crates/switchyard-server/src/main.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/src/main.rs) | Rust | Entry point for the native Switchyard server. |
| [`switchyard/__init__.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard/__init__.py) | Python | Exposes the public Python API (`switchyard.client`, `switchyard.libsy`). |
| [`scripts/run_local_soak_test.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/scripts/run_local_soak_test.py) | Python | Demonstrates how Python scripts interact with the Rust server for testing. |
| [`pyproject.toml`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/pyproject.toml) | TOML (build config) | Declares Python dependencies and build settings for the wheel. |
| [`benchmark/server-configs/tb-lite-single-opus-4-7.toml`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/benchmark/server-configs/tb-lite-single-opus-4-7.toml) | TOML (runtime config) | Example of a routing profile consumed by the Rust server. |

These files work together to illustrate the clear separation of concerns — **Rust** for speed-critical server logic, **Python** for user-facing APIs and orchestration, with TOML-based configuration gluing.

## Summary

- **Switchyard's two primary programming languages** are **Rust** (native server + routing engine) and **Python** (client API + tooling).
- Rust's PyO3 bindings let Python call into the native core without re-serialization overhead.
- Supporting TOML and Bash files handle build config, routing profiles, and CI orchestration.
- Both languages are anchored by real code paths in the GitHub repo — e.g., [`crates/switchyard-server/src/main.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/src/main.rs) and [`switchyard/__init__.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard/__init__.py).

## Frequently Asked Questions

### What is Rust used for in Switchyard?

Rust handles the high-performance HTTP server, protocol translation, routing algorithms, and metrics collection. It compiles into both a standalone server binary and the PyO3 Python extension in `_switchyard_rust`.

### What does the Python API in Switchyard look like?

The Python package is built for building OpenAI/Anthropic decade experience. With `import switchyard`, you create a client, then call `client.Chat`, `client.Completions`, etc., which internally forwards requests to the Rust server over HTTP.

### Do I need to know both Rust and Python to use Switchyard?

No. End users and integration depend on Python. Rust is required only for contributing or when building hardcode-level changes to Switchyard itself. Installing the packaged works fine if you just want the Python API.

### Are optimization tools like TOML configs used in the repo?

Yes, TOML files serve double duty: build metadata (package manifests to build the Rust wheel) and runtime routing definitions (e.g., [`benchmark/server-configs/tb-lite-single-opus-4-7.toml`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/benchmark/server-configs/tb-lite-single-opus-4-7.toml)) that control which upstreams and model profiles a deployed Switchyard server uses.