How to Integrate Switchyard with Existing OpenAI SDK Client Applications

Integrate Switchyard with OpenAI SDK applications by launching its native Rust server and pointing the OPENAI_BASE_URL environment variable to http://localhost:4000/v1—no code changes required.

Switchyard is a lightweight proxy from NVIDIA that sits between your OpenAI SDK clients and actual LLM backends. It rewrites request/response payloads to match the OpenAI wire format, applies intelligent routing, health-aware selection, and optional caching. This guide shows you how to drop Switchyard into existing openai Python code with zero modifications.

What Switchyard Does for OpenAI SDK Integration

Switchyard exposes the standard OpenAI HTTP API at /v1/chat/completions, /v1/completions, and related endpoints. Your existing code continues using the familiar SDK patterns while gaining enterprise routing features.

According to the NVIDIA-NeMo/Switchyard source code, the integration works through three layers:

1. Native Rust server – handles HTTP transport and OpenAI-compatible serialization
2. TOML configuration – defines routes, providers, weights, and caching rules
3. libsy routing algorithm – selects optimal backends based on health, load, and A/B test configuration

Launch the Switchyard Native Server

The entry point for starting the server is switchyard/cli/launch_command.py. The launch_codex function (lines 35-44) invokes launch_codex_config to start the native Rust server:

# switchyard/cli/launch_command.py

# Lines 35-44: CLI entry that starts the native server

Start Switchyard from your terminal:

switchyard launch codex --config routes.toml --model openai/gpt-4

By default, the server listens on http://localhost:4000. The TOML configuration file (conventionally openrouter.toml) defines your provider endpoints, model mappings, and pricing rules.

Configure the OpenAI SDK Base URL

The OpenAI SDK automatically reads OPENAI_BASE_URL from the environment. Point it at Switchyard:

export OPENAI_BASE_URL="http://localhost:4000/v1"
export OPENAI_API_KEY="placeholder"  # Switchyard ignores this value

No other configuration is needed. The SDK sends standard OpenAI JSON payloads to Switchyard, which then handles translation and routing.

Complete Integration Example

import openai
import os

# Start Switchyard separately: switchyard launch codex --config routes.toml

os.environ["OPENAI_BASE_URL"] = "http://localhost:4000/v1"
os.environ["OPENAI_API_KEY"] = "placeholder"

response = openai.ChatCompletion.create(
    model="openai/gpt-4",
    messages=[{"role": "user", "content": "Explain Switchyard routing"}],
)

print(response.choices[0].message["content"])

The model name openai/gpt-4 maps to your configured route. Switchyard's cost_estimator.py (lines 58-69) handles OpenAI pricing semantics and cache behavior lookups.

How Switchyard Translates OpenAI Requests

The translation layer lives in switchyard/cli/launchers/cost_estimator.py. This file defines ModelPriceData entries that match incoming model names against routing rules:

# switchyard/cli/launchers/cost_estimator.py

# Lines 58-69: OpenAI wire format handling and pricing application

For each request, Switchyard:

• Parses the OpenAI-compatible JSON payload
• Looks up the model name in its route table
• Applies any configured caching or cost estimation
• Forwards to the selected provider

Routing and Health-Aware Selection

The core routing algorithm is implemented in Rust within crates/libsy and exposed through Python bindings in switchyard/libsy/algorithms.py:

# switchyard/libsy/algorithms.py

# Entry point for provider selection based on health, load, and A/B testing

When a request arrives, the server queries the libsy algorithm for the optimal route. Factors include:

• Provider health status – automatic failover from unhealthy endpoints
• Load distribution – weighted routing across available capacity
• A/B testing configuration – controlled traffic splitting for model comparisons

Integrating Switchyard with LiteLLM

LiteLLM users can leverage Switchyard's OpenAI-compatible transport by prefixing model names with openai/:

from litellm import completion
import os

os.environ["OPENAI_BASE_URL"] = "http://localhost:4000/v1"
os.environ["OPENAI_API_KEY"] = "any"

response = completion(
    model="openai/gpt-4o-mini",  # Critical: include "openai/" prefix

    messages=[{"role": "user", "content": "Write a haiku about clouds"}],
)

print(response.choices[0].message["content"])

This pattern is demonstrated in examples/experimental/litellm/src/switchyard_litellm/client.py (line 239), where the prefix triggers LiteLLM's OpenAI-compatible transport path.

Custom Routing Configuration

Define sophisticated routing rules in TOML:

# routes.toml

[[model_providers]]
id = "openai"
type = "openai"
base_url = "https://api.openai.com/v1"
api_key = "${OPENAI_API_KEY}"

[[model_providers]]
id = "nvidia"
type = "nvidia"
base_url = "https://api.nvidia.com/v1"
api_key = "${NVIDIA_API_KEY}"

[[routes]]
model = "openai/gpt-4"
provider = "openai"
weight = 0.7

[[routes]]
model = "nvidia/gpt-4"
provider = "nvidia"
weight = 0.3

Launch with:

switchyard launch codex --config routes.toml --model openai/gpt-4

The TOML schema is parsed by the native server and consumed by the libsy routing algorithm.

Key Source Files for OpenAI SDK Integration

File	Purpose
switchyard/cli/launch_command.py	CLI entry points for server launch
switchyard/cli/launchers/codex_cli_launcher.py	switchyard launch codex implementation
switchyard/cli/launchers/cost_estimator.py	OpenAI pricing and cache semantics
switchyard/libsy/algorithms.py	Python bindings for routing algorithm
switchyard_rust/server.py	Native Rust HTTP server
examples/experimental/litellm/src/switchyard_litellm/client.py	LiteLLM integration example

Summary

• Zero code changes: Set OPENAI_BASE_URL=http://localhost:4000/v1 to route existing OpenAI SDK clients through Switchyard
• Native Rust server: Started via switchyard launch codex from switchyard/cli/launch_command.py
• TOML configuration: Define providers, routes, weights, and caching in a declarative format
• Intelligent routing: Health-aware selection via switchyard/libsy/algorithms.py
• LiteLLM compatible: Use openai/ model name prefix for alternative SDK integration

Frequently Asked Questions

Does Switchyard require modifying existing OpenAI SDK code?

No. Switchyard is designed as a transparent proxy. Change the OPENAI_BASE_URL environment variable and optionally OPENAI_API_KEY—all existing openai.ChatCompletion.create() calls work unchanged. The SDK sends identical HTTP payloads to Switchyard instead of OpenAI's servers.

What model naming convention does Switchyard expect?

Switchyard uses provider-prefixed model names like openai/gpt-4 or nvidia/gpt-4. These map to entries in your TOML configuration file. The cost_estimator.py launcher uses these names to look up pricing data and routing rules.

Can Switchyard handle multiple LLM providers simultaneously?

Yes. Define multiple [[model_providers]] blocks in your TOML configuration, then create [[routes]] entries with weights for load balancing. The libsy algorithm in switchyard/libsy/algorithms.py evaluates provider health and configured weights for each request.

Does the OpenAI API key need to be valid when using Switchyard?

No. Switchyard accepts any non-empty value for OPENAI_API_KEY. The proxy injects the actual provider credentials from its configuration file before forwarding requests to backend LLM services.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}