# How to Connect Self-Hosted vLLM Model Servers to Switchyard

> Connect self-hosted vLLM model servers to Switchyard. Configure llm_client, target, and passthrough routes in a TOML file for seamless integration with OpenAI-compatible endpoints.

- Repository: [NVIDIA-NeMo/Switchyard](https://github.com/NVIDIA-NeMo/Switchyard)
- Tags: how-to-guide
- Published: 2026-08-15

---

**Switchyard connects to any OpenAI-compatible endpoint, including self-hosted vLLM servers, by configuring an `llm_client`, `target`, and `passthrough` route in a TOML deployment file.**

Switchyard is an open-source LLM routing gateway from NVIDIA that does not host models itself—it routes traffic to external providers. This guide demonstrates how to integrate a self-hosted vLLM model server with Switchyard using the configuration files and routing logic implemented in `NVIDIA-NeMo/Switchyard`.

## Prerequisites: Start Your vLLM Server

Before configuring Switchyard, launch your vLLM server with an OpenAI-compatible Chat API endpoint. The server must expose the `/v1` path for Switchyard to communicate with it.

```bash
vllm serve ./my-rl-qwen --served-model-name my-rl-qwen --port 8000

```

This command starts vLLM serving your fine-tuned model on port 8000. The `--served-model-name` parameter defines the model identifier that downstream consumers will reference.

As documented in [`docs/routing_algorithms/overview.md`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/docs/routing_algorithms/overview.md), Switchyard's **self-hosted targets** section explicitly supports this pattern for any OpenAI-compatible server, including Ollama and custom implementations.

## Step 1: Declare the LLM Client

Create a TOML configuration file and define an `llm_client` under the **`[llm_clients]`** table. Use `format = "openai_chat"` for chat completions (or `openai` for legacy completions). Point `base_url` to your server's `/v1` endpoint.

For local servers without authentication, **omit `api_key_env`** entirely:

```toml
[llm_clients.local_vllm]
format = "openai_chat"
base_url = "http://localhost:8000/v1"

# no api_key_env needed for local, unauthenticated servers

```

The configuration schema for LLM clients is defined in [`crates/switchyard-server/CONFIGURATION.md`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/CONFIGURATION.md). The `format` parameter tells Switchyard which protocol adapter to use when serializing requests and parsing responses.

## Step 2: Create a Target

Targets bind clients to specific model identifiers. Define a **`[targets]`** entry that references your client by name using the `llm_client` field. The `id` field specifies the model name that your vLLM server expects:

```toml
[targets.local]
id = "my-rl-qwen"
llm_client = "local_vllm"

```

You can add provider-specific options using the **`extra_body`** field. Switchyard performs a shallow merge of these values into each outbound request. This enables vLLM-specific features like `enable_thinking`:

```toml
[targets.local]
id = "my-rl-qwen"
llm_client = "local_vllm"
extra_body = { chat_template_kwargs = { enable_thinking = false } }

```

## Step 3: Define a Passthrough Route

Routes map client-facing model names to targets. For direct proxying without routing logic, use **`type = "passthrough"`** as documented in [`docs/routing_algorithms/overview.md`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/docs/routing_algorithms/overview.md):

```toml
[routes.local]
id = "local"
type = "passthrough"
target = "local"

```

The route's `id` becomes the model name that API clients specify in their requests. In this example, calling Switchyard with `"model": "local"` forwards to the `local` target, which uses the `local_vllm` client to reach your vLLM server.

## Step 4: Start Switchyard

Launch the Switchyard server with your configuration file:

```bash
switchyard-server --config routes.toml --host 127.0.0.1 --port 4000

```

The `switchyard-server` executable is the CLI wrapper implemented in [`switchyard/cli/launchers/native_server.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard/cli/launchers/native_server.py). It loads your TOML deployment and starts the HTTP gateway.

## Step 5: Send Requests Through Switchyard

Query your self-hosted vLLM model via the Switchyard endpoint using the route's `id` as the model name:

```bash
curl http://localhost:4000/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{"model":"local","messages":[{"role":"user","content":"Explain quantum tunneling"}]}'

```

Switchyard performs three operations on each request:
1. **Protocol translation**: Converts the incoming OpenAI format to the target server's expected schema
2. **Request forwarding**: Sends the transformed request to your vLLM server at `http://localhost:8000/v1`
3. **Response translation**: Parses the vLLM response and returns it in OpenAI-compatible format

## Complete Configuration Example

Here is a complete, runnable [`routes.toml`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/routes.toml) for a self-hosted vLLM integration:

```toml
schema_version = 1

[llm_clients.local_vllm]
format = "openai_chat"
base_url = "http://localhost:8000/v1"

[targets.local]
id = "my-rl-qwen"
llm_client = "local_vllm"

[routes.local]
id = "local"
type = "passthrough"
target = "local"

```

## Key Source Files and References

| File | Purpose |
|------|---------|
| [`docs/routing_algorithms/overview.md`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/docs/routing_algorithms/overview.md) | Explains the **self-hosted targets** pattern and `passthrough` route type |
| [`crates/switchyard-server/CONFIGURATION.md`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/CONFIGURATION.md) | Documents **`[llm_clients]`** table, **extra_body** merging, and target configuration |
| [`switchyard/cli/launchers/native_server.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard/cli/launchers/native_server.py) | CLI entry point for `switchyard-server` command |
| [`examples/libsy.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/examples/libsy.py) | Python bindings for embedding Switchyard's Rust routing library in custom applications |

## Summary

- Switchyard routes traffic to **any OpenAI-compatible endpoint**, including self-hosted vLLM servers
- Configure the connection using three TOML sections: **`[llm_clients]`**, **`[targets]`**, and **`[routes]`**
- Use **`format = "openai_chat"`** and point `base_url` to your server's `/v1` endpoint
- Set **`type = "passthrough"`** for direct proxying without routing logic
- Omit **`api_key_env`** for unauthenticated local servers
- Add **`extra_body`** for provider-specific request parameters that Switchyard merges into outbound calls

## Frequently Asked Questions

### Does Switchyard require an API key for self-hosted vLLM servers?

No. Self-hosted servers without authentication do not require an API key. Simply **omit the `api_key_env` field** from your `[llm_clients]` configuration. Switchyard will send requests without authorization headers.

### Can I use Switchyard with Ollama or other local LLM servers?

Yes. Any server implementing the **OpenAI-compatible Chat API** works with Switchyard. Use `format = "openai_chat"` and set `base_url` to your server's `/v1` endpoint. The self-hosted targets pattern in [`docs/routing_algorithms/overview.md`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/docs/routing_algorithms/overview.md) explicitly supports Ollama, vLLM, and custom implementations.

### How do I pass vLLM-specific parameters like `enable_thinking`?

Add them under **`extra_body`** in your target configuration. Switchyard shallow-merges these fields into every outbound request. Example: `extra_body = { chat_template_kwargs = { enable_thinking = false } }`.

### What is the difference between a client, target, and route?

- **Client** (`[llm_clients]`): Defines the protocol adapter and endpoint URL for a backend server
- **Target** (`[targets]`): Binds a client to a specific model ID and optional request modifications
- **Route** (`[routes]`): Maps a client-facing model name to a target, optionally applying routing logic