How to Connect Self-Hosted vLLM Model Servers to Switchyard

Switchyard connects to any OpenAI-compatible endpoint, including self-hosted vLLM servers, by configuring an llm_client, target, and passthrough route in a TOML deployment file.

Switchyard is an open-source LLM routing gateway from NVIDIA that does not host models itself—it routes traffic to external providers. This guide demonstrates how to integrate a self-hosted vLLM model server with Switchyard using the configuration files and routing logic implemented in NVIDIA-NeMo/Switchyard.

Prerequisites: Start Your vLLM Server

Before configuring Switchyard, launch your vLLM server with an OpenAI-compatible Chat API endpoint. The server must expose the /v1 path for Switchyard to communicate with it.

vllm serve ./my-rl-qwen --served-model-name my-rl-qwen --port 8000

This command starts vLLM serving your fine-tuned model on port 8000. The --served-model-name parameter defines the model identifier that downstream consumers will reference.

As documented in docs/routing_algorithms/overview.md, Switchyard's self-hosted targets section explicitly supports this pattern for any OpenAI-compatible server, including Ollama and custom implementations.

Step 1: Declare the LLM Client

Create a TOML configuration file and define an llm_client under the [llm_clients] table. Use format = "openai_chat" for chat completions (or openai for legacy completions). Point base_url to your server's /v1 endpoint.

For local servers without authentication, omit api_key_env entirely:

[llm_clients.local_vllm]
format = "openai_chat"
base_url = "http://localhost:8000/v1"

# no api_key_env needed for local, unauthenticated servers

The configuration schema for LLM clients is defined in crates/switchyard-server/CONFIGURATION.md. The format parameter tells Switchyard which protocol adapter to use when serializing requests and parsing responses.

Step 2: Create a Target

Targets bind clients to specific model identifiers. Define a [targets] entry that references your client by name using the llm_client field. The id field specifies the model name that your vLLM server expects:

[targets.local]
id = "my-rl-qwen"
llm_client = "local_vllm"

You can add provider-specific options using the extra_body field. Switchyard performs a shallow merge of these values into each outbound request. This enables vLLM-specific features like enable_thinking:

[targets.local]
id = "my-rl-qwen"
llm_client = "local_vllm"
extra_body = { chat_template_kwargs = { enable_thinking = false } }

Step 3: Define a Passthrough Route

Routes map client-facing model names to targets. For direct proxying without routing logic, use type = "passthrough" as documented in docs/routing_algorithms/overview.md:

[routes.local]
id = "local"
type = "passthrough"
target = "local"

The route's id becomes the model name that API clients specify in their requests. In this example, calling Switchyard with "model": "local" forwards to the local target, which uses the local_vllm client to reach your vLLM server.

Step 4: Start Switchyard

Launch the Switchyard server with your configuration file:

switchyard-server --config routes.toml --host 127.0.0.1 --port 4000

The switchyard-server executable is the CLI wrapper implemented in switchyard/cli/launchers/native_server.py. It loads your TOML deployment and starts the HTTP gateway.

Step 5: Send Requests Through Switchyard

Query your self-hosted vLLM model via the Switchyard endpoint using the route's id as the model name:

curl http://localhost:4000/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{"model":"local","messages":[{"role":"user","content":"Explain quantum tunneling"}]}'

Switchyard performs three operations on each request:

1. Protocol translation: Converts the incoming OpenAI format to the target server's expected schema
2. Request forwarding: Sends the transformed request to your vLLM server at http://localhost:8000/v1
3. Response translation: Parses the vLLM response and returns it in OpenAI-compatible format

Complete Configuration Example

Here is a complete, runnable routes.toml for a self-hosted vLLM integration:

schema_version = 1

[llm_clients.local_vllm]
format = "openai_chat"
base_url = "http://localhost:8000/v1"

[targets.local]
id = "my-rl-qwen"
llm_client = "local_vllm"

[routes.local]
id = "local"
type = "passthrough"
target = "local"

Key Source Files and References

File	Purpose
docs/routing_algorithms/overview.md	Explains the self-hosted targets pattern and passthrough route type
crates/switchyard-server/CONFIGURATION.md	Documents [llm_clients] table, extra_body merging, and target configuration
switchyard/cli/launchers/native_server.py	CLI entry point for switchyard-server command
examples/libsy.py	Python bindings for embedding Switchyard's Rust routing library in custom applications

Summary

• Switchyard routes traffic to any OpenAI-compatible endpoint, including self-hosted vLLM servers
• Configure the connection using three TOML sections: [llm_clients], [targets], and [routes]
• Use format = "openai_chat" and point base_url to your server's /v1 endpoint
• Set type = "passthrough" for direct proxying without routing logic
• Omit api_key_env for unauthenticated local servers
• Add extra_body for provider-specific request parameters that Switchyard merges into outbound calls

Frequently Asked Questions

Does Switchyard require an API key for self-hosted vLLM servers?

No. Self-hosted servers without authentication do not require an API key. Simply omit the api_key_env field from your [llm_clients] configuration. Switchyard will send requests without authorization headers.

Can I use Switchyard with Ollama or other local LLM servers?

Yes. Any server implementing the OpenAI-compatible Chat API works with Switchyard. Use format = "openai_chat" and set base_url to your server's /v1 endpoint. The self-hosted targets pattern in docs/routing_algorithms/overview.md explicitly supports Ollama, vLLM, and custom implementations.

How do I pass vLLM-specific parameters like enable_thinking?

Add them under extra_body in your target configuration. Switchyard shallow-merges these fields into every outbound request. Example: extra_body = { chat_template_kwargs = { enable_thinking = false } }.

What is the difference between a client, target, and route?

• Client ([llm_clients]): Defines the protocol adapter and endpoint URL for a backend server
• Target ([targets]): Binds a client to a specific model ID and optional request modifications
• Route ([routes]): Maps a client-facing model name to a target, optionally applying routing logic

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}