Programming Languages Used in Switchyard: Rust Core, Python API, and More

TLDR: Switchyard uses Rust for its high-performance native server and routing engine, and Python for its public API, CLI tools, and orchestration scripts — connected via PyO3 bindings, with TOML and Bash handling configuration and CI.

The NVIDIA-NeMo/Switchyard repository is a modern AI gateway that stands out for its deliberate polyglot architecture. Instead of limiting itself to a single language, Switchyard pairs a Rust-based core server with a Python-facing client API, giving developers both low-level performance and high-level ergonomics. As the source tree shows, every performance-critical component lives in the Rust crates/ workspace, while the user-facing switchyard Python package orchestrates workflows and exposes the OpenAI-compatible interface. This article breaks down exactly which languages appear, where they live in the codebase, and why the split matters for developers.

The Two Primary Languages in Switchyard

Language	Role in Switchyard	Representative Source
Rust	Implements the high-performance native server, routing algorithms, protocol types, and the PyO3 bindings that expose the Rust core to Python.	crates/switchyard-py/Cargo.toml – defines the Rust crate that builds the Python bindings.
Python	Provides the public API (switchyard package), CLI utilities, integration scripts, and higher-level workflow orchestration.	switchyard/init.py – entry point for the Python library.

This composition is by design: as implemented in Switchyard, Rust supplies deterministic, low-latency processing while Python gives developers a familiar scripting interface and deep ecosystem integration.

The Rust Core: Performance Where It Counts

The crates/ directory at the repository root contains a workspace of Rust crates — switchyard-libsy, switchyard-protocol, switchyard-server, switchyard-soak, and others. These crates compile into two artifacts:

• A native binary (switchyard-server) that can run standalone, handling HTTP routing directly.
• A shared library (_switchyard_rust) that Python loads at runtime through PyO3.

On the Rust side, Switchyard handles high-throughput HTTP routing, protocol translation, metrics collection, and A/B testing. All of these operations run without the overhead of the Python interpreter, which keeps latency low even under heavy concurrent load.

The Python Layer: Developer-Facing Convenience

The switchyard package offers a provider-neutral client API that mirrors the OpenAI and Anthropic SDKs. It exposes builder methods like client.Chat and client.Completions, and at runtime it loads the compiled Rust extension (_switchyard_rust) automatically. Helper scripts under scripts/ and examples/ are all written in Python, simplifying local testing, benchmark orchestration, and CI integration.

How Rust and Python Work Together

The key glue is the PyO3 binding in crates/switchyard-py/Cargo.toml. This Rust crate compiles into a CPython extension module. When you install Switchyard from source or via a wheel, the build generates the native extension that the Python package imports. Effectively, you get:

1. A fast Rust skeleton that lazy-optimizes and offloads heavy traffic.
2. A comfortable Python surface that hides the low-level server complexity.

This mirrors common ecosystem patterns (like pydantic-core in Rust or ChromaDB), but in Switchyard it is baked into the routing and benchmark tooling rather than just data validation.

Code Examples in Both Languages

Python: Basic Switchyard Client

import switchyard

# Create a synchronous client that talks to a locally-running Switchyard server

client = switchyard.client.SyncClient(base_url="http://localhost:4000")

# Send a simple chat request

response = client.chat(messages=[{"role": "user", "content": "Hello, Switchyard!"}])

print("Switchyard reply:", response["choices"][0]["message"]["content"])

The switchyard.client module lives in the Python package and forwards the request to the Rust server via the PyO3 binding.

Rust: Minimal Server Entry Point (Excerpt)

// crates/switchyard-server/src/main.rs
use switchyard_server::Server;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Bind to the address supplied via CLI arguments
    let server = Server::bind("0.0.0.0:4000").await?;
    // Start handling incoming LLM traffic
    server.serve().await?;
    Ok(())
}

The server is compiled from the Rust crate switchyard-server and runs as a native binary, delivering the routing logic used by the Python client.

Supporting Languages and Configuration Files

While Rust and Python are the two programming languages, Switchyard also relies on two auxiliary file formats for build and runtime configuration:

1. TOML – Used for manifest files (Cargo.toml, pyproject.toml) and for custom route descriptors like benchmark/server-configs/tb-lite-single-opus-4-7.toml. These define which model providers, latency targets, and traffic splits the Rust server should honor.

2. Bash / Shell – Scripts such as benchmark/run-baseline.sh orchestrate micro-benchmark runs, A/B pipelines, and CI triggering.

These aren't programming languages in the traditional sense, but they are essential support files that ensure the whole system builds and runs as one unit.

Where to Find Them: Key Files

File	Language	Why it matters
crates/switchyard-py/Cargo.toml	Rust (manifest)	Shows the Rust crate that builds the Python bindings (pyo3 dependency).
crates/switchyard-server/src/main.rs	Rust	Entry point for the native Switchyard server.
switchyard/__init__.py	Python	Exposes the public Python API (switchyard.client, switchyard.libsy).
scripts/run_local_soak_test.py	Python	Demonstrates how Python scripts interact with the Rust server for testing.
pyproject.toml	TOML (build config)	Declares Python dependencies and build settings for the wheel.
benchmark/server-configs/tb-lite-single-opus-4-7.toml	TOML (runtime config)	Example of a routing profile consumed by the Rust server.

These files work together to illustrate the clear separation of concerns — Rust for speed-critical server logic, Python for user-facing APIs and orchestration, with TOML-based configuration gluing.

Summary

• Switchyard's two primary programming languages are Rust (native server + routing engine) and Python (client API + tooling).
• Rust's PyO3 bindings let Python call into the native core without re-serialization overhead.
• Supporting TOML and Bash files handle build config, routing profiles, and CI orchestration.
• Both languages are anchored by real code paths in the GitHub repo — e.g., crates/switchyard-server/src/main.rs and switchyard/__init__.py.

Frequently Asked Questions

What is Rust used for in Switchyard?

Rust handles the high-performance HTTP server, protocol translation, routing algorithms, and metrics collection. It compiles into both a standalone server binary and the PyO3 Python extension in _switchyard_rust.

What does the Python API in Switchyard look like?

The Python package is built for building OpenAI/Anthropic decade experience. With import switchyard, you create a client, then call client.Chat, client.Completions, etc., which internally forwards requests to the Rust server over HTTP.

Do I need to know both Rust and Python to use Switchyard?

No. End users and integration depend on Python. Rust is required only for contributing or when building hardcode-level changes to Switchyard itself. Installing the packaged works fine if you just want the Python API.

Are optimization tools like TOML configs used in the repo?

Yes, TOML files serve double duty: build metadata (package manifests to build the Rust wheel) and runtime routing definitions (e.g., benchmark/server-configs/tb-lite-single-opus-4-7.toml) that control which upstreams and model profiles a deployed Switchyard server uses.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}