# How to Integrate Switchyard with Existing OpenAI SDK Client Applications

> Effortlessly integrate Switchyard with your OpenAI SDK apps. Launch the Rust server and set OPENAI_BASE_URL to localhost:4000/v1. No code modifications needed for seamless integration.

- Repository: [NVIDIA-NeMo/Switchyard](https://github.com/NVIDIA-NeMo/Switchyard)
- Tags: how-to-guide
- Published: 2026-08-15

---

**Integrate Switchyard with OpenAI SDK applications by launching its native Rust server and pointing the `OPENAI_BASE_URL` environment variable to `http://localhost:4000/v1`—no code changes required.**

Switchyard is a lightweight proxy from NVIDIA that sits between your **OpenAI SDK** clients and actual LLM backends. It rewrites request/response payloads to match the OpenAI wire format, applies intelligent routing, health-aware selection, and optional caching. This guide shows you how to drop Switchyard into existing `openai` Python code with zero modifications.

## What Switchyard Does for OpenAI SDK Integration

Switchyard exposes the standard OpenAI HTTP API at `/v1/chat/completions`, `/v1/completions`, and related endpoints. Your existing code continues using the familiar SDK patterns while gaining enterprise routing features.

According to the **NVIDIA-NeMo/Switchyard** source code, the integration works through three layers:

1. **Native Rust server** – handles HTTP transport and OpenAI-compatible serialization
2. **TOML configuration** – defines routes, providers, weights, and caching rules
3. **libsy routing algorithm** – selects optimal backends based on health, load, and A/B test configuration

## Launch the Switchyard Native Server

The entry point for starting the server is [`switchyard/cli/launch_command.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard/cli/launch_command.py). The `launch_codex` function (lines 35-44) invokes `launch_codex_config` to start the native Rust server:

```python

# switchyard/cli/launch_command.py

# Lines 35-44: CLI entry that starts the native server

```

Start Switchyard from your terminal:

```bash
switchyard launch codex --config routes.toml --model openai/gpt-4

```

By default, the server listens on `http://localhost:4000`. The TOML configuration file (conventionally [`openrouter.toml`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/openrouter.toml)) defines your provider endpoints, model mappings, and pricing rules.

## Configure the OpenAI SDK Base URL

The **OpenAI SDK** automatically reads `OPENAI_BASE_URL` from the environment. Point it at Switchyard:

```bash
export OPENAI_BASE_URL="http://localhost:4000/v1"
export OPENAI_API_KEY="placeholder"  # Switchyard ignores this value

```

No other configuration is needed. The SDK sends standard OpenAI JSON payloads to Switchyard, which then handles translation and routing.

## Complete Integration Example

```python
import openai
import os

# Start Switchyard separately: switchyard launch codex --config routes.toml

os.environ["OPENAI_BASE_URL"] = "http://localhost:4000/v1"
os.environ["OPENAI_API_KEY"] = "placeholder"

response = openai.ChatCompletion.create(
    model="openai/gpt-4",
    messages=[{"role": "user", "content": "Explain Switchyard routing"}],
)

print(response.choices[0].message["content"])

```

The model name `openai/gpt-4` maps to your configured route. Switchyard's [`cost_estimator.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/cost_estimator.py) (lines 58-69) handles OpenAI pricing semantics and cache behavior lookups.

## How Switchyard Translates OpenAI Requests

The translation layer lives in [`switchyard/cli/launchers/cost_estimator.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard/cli/launchers/cost_estimator.py). This file defines `ModelPriceData` entries that match incoming model names against routing rules:

```python

# switchyard/cli/launchers/cost_estimator.py

# Lines 58-69: OpenAI wire format handling and pricing application

```

For each request, Switchyard:

- Parses the OpenAI-compatible JSON payload
- Looks up the model name in its route table
- Applies any configured caching or cost estimation
- Forwards to the selected provider

## Routing and Health-Aware Selection

The core routing algorithm is implemented in Rust within `crates/libsy` and exposed through Python bindings in [`switchyard/libsy/algorithms.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard/libsy/algorithms.py):

```python

# switchyard/libsy/algorithms.py

# Entry point for provider selection based on health, load, and A/B testing

```

When a request arrives, the server queries the libsy algorithm for the optimal route. Factors include:

- **Provider health status** – automatic failover from unhealthy endpoints
- **Load distribution** – weighted routing across available capacity
- **A/B testing configuration** – controlled traffic splitting for model comparisons

## Integrating Switchyard with LiteLLM

LiteLLM users can leverage Switchyard's OpenAI-compatible transport by prefixing model names with `openai/`:

```python
from litellm import completion
import os

os.environ["OPENAI_BASE_URL"] = "http://localhost:4000/v1"
os.environ["OPENAI_API_KEY"] = "any"

response = completion(
    model="openai/gpt-4o-mini",  # Critical: include "openai/" prefix

    messages=[{"role": "user", "content": "Write a haiku about clouds"}],
)

print(response.choices[0].message["content"])

```

This pattern is demonstrated in [`examples/experimental/litellm/src/switchyard_litellm/client.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/examples/experimental/litellm/src/switchyard_litellm/client.py) (line 239), where the prefix triggers LiteLLM's OpenAI-compatible transport path.

## Custom Routing Configuration

Define sophisticated routing rules in TOML:

```toml

# routes.toml

[[model_providers]]
id = "openai"
type = "openai"
base_url = "https://api.openai.com/v1"
api_key = "${OPENAI_API_KEY}"

[[model_providers]]
id = "nvidia"
type = "nvidia"
base_url = "https://api.nvidia.com/v1"
api_key = "${NVIDIA_API_KEY}"

[[routes]]
model = "openai/gpt-4"
provider = "openai"
weight = 0.7

[[routes]]
model = "nvidia/gpt-4"
provider = "nvidia"
weight = 0.3

```

Launch with:

```bash
switchyard launch codex --config routes.toml --model openai/gpt-4

```

The TOML schema is parsed by the native server and consumed by the libsy routing algorithm.

## Key Source Files for OpenAI SDK Integration

| File | Purpose |
|------|---------|
| [`switchyard/cli/launch_command.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard/cli/launch_command.py) | CLI entry points for server launch |
| [`switchyard/cli/launchers/codex_cli_launcher.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard/cli/launchers/codex_cli_launcher.py) | `switchyard launch codex` implementation |
| [`switchyard/cli/launchers/cost_estimator.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard/cli/launchers/cost_estimator.py) | OpenAI pricing and cache semantics |
| [`switchyard/libsy/algorithms.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard/libsy/algorithms.py) | Python bindings for routing algorithm |
| [`switchyard_rust/server.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard_rust/server.py) | Native Rust HTTP server |
| [`examples/experimental/litellm/src/switchyard_litellm/client.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/examples/experimental/litellm/src/switchyard_litellm/client.py) | LiteLLM integration example |

## Summary

- **Zero code changes**: Set `OPENAI_BASE_URL=http://localhost:4000/v1` to route existing OpenAI SDK clients through Switchyard
- **Native Rust server**: Started via `switchyard launch codex` from [`switchyard/cli/launch_command.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard/cli/launch_command.py)
- **TOML configuration**: Define providers, routes, weights, and caching in a declarative format
- **Intelligent routing**: Health-aware selection via [`switchyard/libsy/algorithms.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard/libsy/algorithms.py)
- **LiteLLM compatible**: Use `openai/` model name prefix for alternative SDK integration

## Frequently Asked Questions

### Does Switchyard require modifying existing OpenAI SDK code?

No. Switchyard is designed as a transparent proxy. Change the `OPENAI_BASE_URL` environment variable and optionally `OPENAI_API_KEY`—all existing `openai.ChatCompletion.create()` calls work unchanged. The SDK sends identical HTTP payloads to Switchyard instead of OpenAI's servers.

### What model naming convention does Switchyard expect?

Switchyard uses provider-prefixed model names like `openai/gpt-4` or `nvidia/gpt-4`. These map to entries in your TOML configuration file. The [`cost_estimator.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/cost_estimator.py) launcher uses these names to look up pricing data and routing rules.

### Can Switchyard handle multiple LLM providers simultaneously?

Yes. Define multiple `[[model_providers]]` blocks in your TOML configuration, then create `[[routes]]` entries with weights for load balancing. The libsy algorithm in [`switchyard/libsy/algorithms.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard/libsy/algorithms.py) evaluates provider health and configured weights for each request.

### Does the OpenAI API key need to be valid when using Switchyard?

No. Switchyard accepts any non-empty value for `OPENAI_API_KEY`. The proxy injects the actual provider credentials from its configuration file before forwarding requests to backend LLM services.