How to Manage Multiple Concurrent Routes and Targets in a Single Switchyard Deployment

Switchyard lets you define unlimited LLM clients, targets, and routes in one TOML file, with all routes served concurrently by a single HTTP server process that selects targets at request time based on configurable routing algorithms.

A Switchyard deployment is fundamentally a declarative configuration defined in a TOML file. Within one deployment, you can declare multiple LLM clients (provider connections), targets (concrete upstream models), and routes (model IDs exposed to callers). Routes can use passthrough for single targets or algorithms like random, llm_classifier, or stage_router to distribute traffic across multiple targets. The Axum-based HTTP server in [switchyard-server/src/lib.rs](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/src/lib.rs) registers endpoints once and handles all routes concurrently, extracting the model field from each request to determine which routing logic to apply.

Understanding the Deployment Hierarchy

Switchyard organizes configuration into three layers. Mastering this hierarchy is essential for managing multiple concurrent routes and targets effectively.

LLM Clients: Provider Connections

LLM clients define how Switchyard communicates with upstream providers—specifying the request format, base URL, and credential source.

[llm_clients.openrouter]
format = "openai_chat"
base_url = "https://openrouter.ai/api/v1"
api_key_env = "OPENROUTER_API_KEY"

You can declare multiple clients in the same deployment to blend models from OpenAI, Anthropic, local vLLM instances, or any OpenAI-compatible endpoint.

Targets: Concrete Model Bindings

Targets bind a specific model ID to an LLM client. This indirection lets you reference the same underlying model through different configurations or aliases.

[targets.strong]
id = "openai/gpt-4o"
llm_client = "openrouter"

[targets.weak]
id = "openai/gpt-4o-mini"
llm_client = "openrouter"

[targets.local]
id = "my-rl-qwen"
llm_client = "local_vllm"

Routes: Exposed Model IDs with Routing Logic

Routes are what callers actually request. Each route specifies:

• id: The model name callers use in their model field
• type: Either passthrough or a routing algorithm
• target/targets: Where to send requests

Configuring Concurrent Routes: A Complete Example

Here's a production-ready routes.toml demonstrating three simultaneous routing strategies:

schema_version = 1

# ---------- LLM clients ----------

[llm_clients.openrouter]
format = "openai_chat"
base_url = "https://openrouter.ai/api/v1"
api_key_env = "OPENROUTER_API_KEY"

# ---------- Targets ----------

[targets.strong]
id = "openai/gpt-4o"
llm_client = "openrouter"

[targets.weak]
id = "openai/gpt-4o-mini"
llm_client = "openrouter"

[targets.local]
id = "my-rl-qwen"
llm_client = "local_vllm"

# ---------- Routes ----------

# Passthrough: always uses the weak model

[routes.fast]
id = "fast"
type = "passthrough"
target = "weak"

# Random A/B test with weighted traffic split

[routes.smart]
id = "smart"
type = "random"
targets = ["strong", "weak"]
weights = [3, 7]  # 30% strong, 70% weak

# Stage router: classifier-driven routing with fallback

[routes.assist]
id = "assist"
type = "stage_router"
targets = ["weak", "strong"]
policy = "default"

How Concurrent Route Handling Works Internally

When you start the server, the following initialization sequence executes:

1. TOML parsing: The configuration is loaded and validated
2. ServerState construction: ServerState holds all routes, targets, and clients in memory
3. Router registration: build_switchyard_router creates an Axum router with endpoints registered once for all routes (/v1/chat/completions, /v1/models, etc.)

At request time, the flow is:

Request → HTTP server → extract "model" field → lookup route → execute routing algorithm → call selected target → return response

The routing decision happens inside handle_endpoint, not at the HTTP layer. This design means:

• No port proliferation: All routes share one HTTP server
• Algorithm flexibility: Each route runs its own selection logic
• Independent metrics: Per-route observability built-in

Starting the Server with Multiple Routes

Launch with your configuration file:

export OPENROUTER_API_KEY="your-openrouter-key"
switchyard-server --config routes.toml --host 127.0.0.1 --port 4000

The server immediately serves all configured routes concurrently.

Sending Requests to Different Routes

# Fast route – deterministic passthrough to weak model

curl http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"fast","messages":[{"role":"user","content":"Hello"}]}'

# Smart route – probabilistic split between models

curl http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"smart","messages":[{"role":"user","content":"Explain quantum computing"}]}'

# Assist route – classifier may invoke multiple targets sequentially

curl http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"assist","messages":[{"role":"user","content":"Write a poem about AI"}]}'

Route-Specific Algorithms and When to Use Them

Algorithm	Use Case	Configuration
passthrough	Deterministic routing, cost testing, baseline models	Single target field
random	A/B testing, canary deployments, cost optimization	targets array + weights
llm_classifier	Intent-based routing using a classifier model	classifier_target + label mappings
stage_router	Multi-stage inference, fallback chains, speculative execution	policy + ordered targets

For algorithm-specific configuration, consult [docs/routing_algorithms/random_routing.md](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/docs/routing_algorithms/random_routing.md) and [docs/routing_algorithms/stage_router_routing.md](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/docs/routing_algorithms/stage_router_routing.md).

Monitoring Concurrent Routes and Targets

Switchyard exposes per-route metrics automatically:

# Health and readiness

curl http://localhost:4000/health

# Prometheus metrics with route labels

curl http://localhost:4000/metrics

# Detailed routing decisions (with routing_log enabled)

curl http://localhost:4000/v1/routing/session-stats

The stats_observer module tracks request counts, latencies, and routing outcomes separately for each route.

Adding or Modifying Routes Without Code Changes

Switchyard's architecture separates declarative configuration from imperative code. To add routes or targets:

1. Edit your routes.toml
2. Restart the server (or use a process manager with hot reload)

No Rust or Python changes required. This enables rapid experimentation:

• Add a new target pointing to a fine-tuned model
• Clone an existing route with different weights for a canary test
• Introduce a stage router to optimize latency on cached queries

Key Implementation Files

Understanding these source files helps when debugging multi-route deployments:

• [crates/switchyard-server/src/lib.rs](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/src/lib.rs): ServerState, build_switchyard_router, and request dispatch
• [crates/switchyard-server/tests/server.rs](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/tests/server.rs): Integration tests showing random, classifier, and stage routing
• [docs/routing_algorithms/overview.md](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/docs/routing_algorithms/overview.md): Conceptual documentation for route/target relationships

Summary

• Single TOML file: Declare unlimited clients, targets, and routes in one configuration
• Unified HTTP server: One Axum process serves all routes via build_switchyard_router
• Runtime routing: The model field in each request triggers the appropriate algorithm
• Algorithm variety: Use passthrough, random, llm_classifier, stage_router, or custom implementations per route
• Zero-code changes: Add, remove, or modify routes and targets by editing configuration only
• Built-in observability: Per-route Prometheus metrics and session statistics via stats_observer

Frequently Asked Questions

How many routes can I run in one Switchyard deployment?

There is no hard limit enforced by the server. The practical limit depends on memory for storing route configurations and the performance characteristics of your targets. The Axum router registers endpoints once regardless of route count, so HTTP overhead remains constant. Test configurations with dozens of routes run successfully; monitor memory usage and response latency as you scale.

Can different routes use different LLM providers?

Yes. Each target binds to a specific LLM client, and each client configures a distinct provider. You can route simultaneously to OpenAI, Anthropic, local vLLM, and any OpenAI-compatible endpoint from the same deployment.

What happens if I request a model ID that isn't configured?

The server returns an HTTP 400 error indicating an unknown model. The routing lookup occurs after JSON parsing but before any upstream call, so no provider credentials are consumed for invalid model names.

Does Switchyard support hot reloading of route configuration?

The core server compiles configuration at startup into an immutable ServerState. For zero-downtime updates, run Switchyard behind a process manager or load balancer that can drain and restart instances with new configuration. The lightweight startup time makes this practical for production deployments.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}