# How to Manage Multiple Concurrent Routes and Targets in a Single Switchyard Deployment

> Learn to manage multiple concurrent routes and targets in a single Switchyard deployment. Define unlimited LLM clients and routes in one TOML file for efficient HTTP server operations.

- Repository: [NVIDIA-NeMo/Switchyard](https://github.com/NVIDIA-NeMo/Switchyard)
- Tags: how-to-guide
- Published: 2026-08-15

---

**Switchyard lets you define unlimited LLM clients, targets, and routes in one TOML file, with all routes served concurrently by a single HTTP server process that selects targets at request time based on configurable routing algorithms.**

A Switchyard **deployment** is fundamentally a declarative configuration defined in a TOML file. Within one deployment, you can declare multiple **LLM clients** (provider connections), **targets** (concrete upstream models), and **routes** (model IDs exposed to callers). Routes can use **passthrough** for single targets or algorithms like **random**, **llm_classifier**, or **stage_router** to distribute traffic across multiple targets. The Axum-based HTTP server in [[`switchyard-server/src/lib.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard-server/src/lib.rs)](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/src/lib.rs) registers endpoints once and handles all routes concurrently, extracting the `model` field from each request to determine which routing logic to apply.

## Understanding the Deployment Hierarchy

Switchyard organizes configuration into three layers. Mastering this hierarchy is essential for managing multiple concurrent routes and targets effectively.

### LLM Clients: Provider Connections

**LLM clients** define how Switchyard communicates with upstream providers—specifying the request format, base URL, and credential source.

```toml
[llm_clients.openrouter]
format = "openai_chat"
base_url = "https://openrouter.ai/api/v1"
api_key_env = "OPENROUTER_API_KEY"

```

You can declare multiple clients in the same deployment to blend models from OpenAI, Anthropic, local vLLM instances, or any OpenAI-compatible endpoint.

### Targets: Concrete Model Bindings

**Targets** bind a specific model ID to an LLM client. This indirection lets you reference the same underlying model through different configurations or aliases.

```toml
[targets.strong]
id = "openai/gpt-4o"
llm_client = "openrouter"

[targets.weak]
id = "openai/gpt-4o-mini"
llm_client = "openrouter"

[targets.local]
id = "my-rl-qwen"
llm_client = "local_vllm"

```

### Routes: Exposed Model IDs with Routing Logic

**Routes** are what callers actually request. Each route specifies:

- **`id`**: The model name callers use in their `model` field
- **`type`**: Either `passthrough` or a routing algorithm
- **`target`/`targets`**: Where to send requests

## Configuring Concurrent Routes: A Complete Example

Here's a production-ready [`routes.toml`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/routes.toml) demonstrating three simultaneous routing strategies:

```toml
schema_version = 1

# ---------- LLM clients ----------

[llm_clients.openrouter]
format = "openai_chat"
base_url = "https://openrouter.ai/api/v1"
api_key_env = "OPENROUTER_API_KEY"

# ---------- Targets ----------

[targets.strong]
id = "openai/gpt-4o"
llm_client = "openrouter"

[targets.weak]
id = "openai/gpt-4o-mini"
llm_client = "openrouter"

[targets.local]
id = "my-rl-qwen"
llm_client = "local_vllm"

# ---------- Routes ----------

# Passthrough: always uses the weak model

[routes.fast]
id = "fast"
type = "passthrough"
target = "weak"

# Random A/B test with weighted traffic split

[routes.smart]
id = "smart"
type = "random"
targets = ["strong", "weak"]
weights = [3, 7]  # 30% strong, 70% weak

# Stage router: classifier-driven routing with fallback

[routes.assist]
id = "assist"
type = "stage_router"
targets = ["weak", "strong"]
policy = "default"

```

## How Concurrent Route Handling Works Internally

When you start the server, the following initialization sequence executes:

1. **TOML parsing**: The configuration is loaded and validated
2. **ServerState construction**: [`ServerState`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/src/lib.rs#L159-L170) holds all routes, targets, and clients in memory
3. **Router registration**: [`build_switchyard_router`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/src/lib.rs#L998-L1014) creates an Axum router with endpoints registered **once** for all routes (`/v1/chat/completions`, `/v1/models`, etc.)

At request time, the flow is:

```

Request → HTTP server → extract "model" field → lookup route → execute routing algorithm → call selected target → return response

```

The routing decision happens inside [`handle_endpoint`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/src/lib.rs), not at the HTTP layer. This design means:

- **No port proliferation**: All routes share one HTTP server
- **Algorithm flexibility**: Each route runs its own selection logic
- **Independent metrics**: Per-route observability built-in

## Starting the Server with Multiple Routes

Launch with your configuration file:

```bash
export OPENROUTER_API_KEY="your-openrouter-key"
switchyard-server --config routes.toml --host 127.0.0.1 --port 4000

```

The server immediately serves all configured routes concurrently.

## Sending Requests to Different Routes

```bash

# Fast route – deterministic passthrough to weak model

curl http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"fast","messages":[{"role":"user","content":"Hello"}]}'

# Smart route – probabilistic split between models

curl http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"smart","messages":[{"role":"user","content":"Explain quantum computing"}]}'

# Assist route – classifier may invoke multiple targets sequentially

curl http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"assist","messages":[{"role":"user","content":"Write a poem about AI"}]}'

```

## Route-Specific Algorithms and When to Use Them

| Algorithm | Use Case | Configuration |
|-----------|----------|---------------|
| **`passthrough`** | Deterministic routing, cost testing, baseline models | Single `target` field |
| **`random`** | A/B testing, canary deployments, cost optimization | `targets` array + `weights` |
| **`llm_classifier`** | Intent-based routing using a classifier model | `classifier_target` + label mappings |
| **`stage_router`** | Multi-stage inference, fallback chains, speculative execution | `policy` + ordered `targets` |

For algorithm-specific configuration, consult [[`docs/routing_algorithms/random_routing.md`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/docs/routing_algorithms/random_routing.md)](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/docs/routing_algorithms/random_routing.md) and [[`docs/routing_algorithms/stage_router_routing.md`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/docs/routing_algorithms/stage_router_routing.md)](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/docs/routing_algorithms/stage_router_routing.md).

## Monitoring Concurrent Routes and Targets

Switchyard exposes per-route metrics automatically:

```bash

# Health and readiness

curl http://localhost:4000/health

# Prometheus metrics with route labels

curl http://localhost:4000/metrics

# Detailed routing decisions (with routing_log enabled)

curl http://localhost:4000/v1/routing/session-stats

```

The [`stats_observer`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/src/lib.rs#L49-L86) module tracks request counts, latencies, and routing outcomes separately for each route.

## Adding or Modifying Routes Without Code Changes

Switchyard's architecture separates **declarative configuration** from **imperative code**. To add routes or targets:

1. Edit your [`routes.toml`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/routes.toml)
2. Restart the server (or use a process manager with hot reload)

No Rust or Python changes required. This enables rapid experimentation:

- Add a new target pointing to a fine-tuned model
- Clone an existing route with different weights for a canary test
- Introduce a stage router to optimize latency on cached queries

## Key Implementation Files

Understanding these source files helps when debugging multi-route deployments:

- **[[`crates/switchyard-server/src/lib.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/src/lib.rs)](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/src/lib.rs)**: `ServerState`, `build_switchyard_router`, and request dispatch
- **[[`crates/switchyard-server/tests/server.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/tests/server.rs)](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/tests/server.rs)**: Integration tests showing random, classifier, and stage routing
- **[[`docs/routing_algorithms/overview.md`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/docs/routing_algorithms/overview.md)](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/docs/routing_algorithms/overview.md)**: Conceptual documentation for route/target relationships

## Summary

- **Single TOML file**: Declare unlimited clients, targets, and routes in one configuration
- **Unified HTTP server**: One Axum process serves all routes via [`build_switchyard_router`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/src/lib.rs#L998-L1014)
- **Runtime routing**: The `model` field in each request triggers the appropriate algorithm
- **Algorithm variety**: Use `passthrough`, `random`, `llm_classifier`, `stage_router`, or custom implementations per route
- **Zero-code changes**: Add, remove, or modify routes and targets by editing configuration only
- **Built-in observability**: Per-route Prometheus metrics and session statistics via [`stats_observer`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/src/lib.rs#L49-L86)

## Frequently Asked Questions

### How many routes can I run in one Switchyard deployment?

There is no hard limit enforced by the server. The practical limit depends on memory for storing route configurations and the performance characteristics of your targets. The Axum router registers endpoints once regardless of route count, so HTTP overhead remains constant. Test configurations with dozens of routes run successfully; monitor memory usage and response latency as you scale.

### Can different routes use different LLM providers?

Yes. Each target binds to a specific LLM client, and each client configures a distinct provider. You can route simultaneously to OpenAI, Anthropic, local vLLM, and any OpenAI-compatible endpoint from the same deployment.

### What happens if I request a model ID that isn't configured?

The server returns an HTTP 400 error indicating an unknown model. The routing lookup occurs after JSON parsing but before any upstream call, so no provider credentials are consumed for invalid model names.

### Does Switchyard support hot reloading of route configuration?

The core server compiles configuration at startup into an immutable `ServerState`. For zero-downtime updates, run Switchyard behind a process manager or load balancer that can drain and restart instances with new configuration. The lightweight startup time makes this practical for production deployments.