How to Configure Weighted Random Routing for A/B Testing in Switchyard TOML Deployments

Assign a weights array parallel to your targets array in the [routes.<name>] section to split traffic proportionally across models for reproducible A/B testing.

Switchyard's random routing algorithm enables traffic splitting across multiple model targets without prompt inspection. According to the NVIDIA-NeMo/Switchyard source code, this approach is ideal for benchmark comparisons, staged rollouts, and controlled experiments where you need precise control over request distribution.

Understanding Weighted Random Routing in Switchyard

The random router selects a target independently for every request. When you provide weights, Switchyard normalizes them into probabilities. Omitting weights triggers equal distribution across all targets.

The routing logic lives in crates/libsy/src/algorithms/random.rs. At startup, the server validates that:

• Every weight is a finite, non‑negative number
• At least one weight is greater than zero

The optional seed parameter ensures deterministic selection order—critical for reproducible test suites.

Required TOML Schema Elements

Configure weighted random routing under [routes.<name>] with these keys:

Key	Required	Description
type	Yes	Must be "random"
targets	Yes	Ordered array of target names defined in [targets.*] sections
weights	No	Ordered array of relative weights matching targets order
seed	No	Integer seed for reproducible random selection

The TOML schema is documented in docs/reference/toml_schema.md (lines 30–34) and the detailed usage guide appears in docs/routing_algorithms/random_routing.md.

Basic Configuration: 30/70 A/B Split

This example routes 30% of traffic to GPT-4o and 70% to GPT-4o-mini:

schema_version = 1

[llm_clients.openrouter]
format = "openai_chat"
base_url = "https://openrouter.ai/api/v1"
api_key_env = "OPENROUTER_API_KEY"

[targets.strong]
id = "openai/gpt-4o"
llm_client = "openrouter"

[targets.weak]
id = "openai/gpt-4o-mini"
llm_client = "openrouter"

[routes.ab_test]
id = "ab-test"
type = "random"
targets = ["strong", "weak"]
weights = [3, 7]          # 3:7 ratio → ~30% / 70%

seed = 42                 # reproducible selection order

The weights 3 and 7 are relative—Switchyard computes probabilities as 3/10 = 0.3 and 7/10 = 0.7. Any proportional values work (e.g., [30, 70] or [0.3, 0.7]).

Equal-Weight A/B Test: Omit Weights

For a 50/50 split, omit the weights key entirely:

[routes.ab_test]
id = "ab-test"
type = "random"
targets = ["strong", "weak"]

# No weights → equal 50/50 distribution

This matches the pattern found in dev-server/config.toml for the [routes.random] definition.

Temporarily Disabling a Target

Set a target's weight to zero to stop routing traffic without removing the configuration:

[routes.ab_test]
id = "ab-test"
type = "random"
targets = ["strong", "weak"]
weights = [1, 0]          # 100% to "strong", 0% to "weak"

Useful for maintenance windows or emergency rollbacks.

Validating and Running Your Deployment

Start the Switchyard server with your TOML configuration:

export OPENROUTER_API_KEY="your-openrouter-key"

# Validate configuration without starting services

switchyard-server --config routes.toml --dry-run

# Start the server

switchyard-server --config routes.toml --host 127.0.0.1 --port 4000

Send requests through the route ID (ab-test in these examples):

curl http://localhost:4000/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{"model":"ab-test","messages":[{"role":"user","content":"Hello"}]}'

Each request triggers independent weighted selection. With seed = 42, the sequence is deterministic across server restarts.

Key Source Files

File	Purpose
crates/libsy/src/algorithms/random.rs	Rust implementation of weighted random selector
docs/routing_algorithms/random_routing.md	Usage guide with weighted examples and seed behavior
docs/reference/toml_schema.md	Formal TOML schema for [routes.<name>] including weights
dev-server/config.toml	Working example of random route configuration
examples/experimental/litellm/README.md	Integration with LiteLLM gateway (experimental)

Summary

• weights array controls proportional traffic split; omitted means equal weights
• Validation requires finite, non-negative values with at least one positive weight
• seed enables reproducible test sequences across restarts
• Zero weights effectively disable targets without configuration changes
• Core algorithm implemented in crates/libsy/src/algorithms/random.rs

Frequently Asked Questions

What happens if weights don't sum to 1 or 100?

Switchyard normalizes weights automatically. Values [3, 7], [0.3, 0.7], and [30, 70] produce identical 30%/70% splits. The algorithm divides each weight by the total sum.

Can I use weighted random routing with more than two targets?

Yes. The targets and weights arrays accept any matching length. Add additional target names and corresponding weights for multi-variant tests.

Does the seed guarantee the same model answers the same request?

No. The seed makes the selection sequence deterministic, not request-to-model mapping. Two identical requests in sequence may still hit different models due to independent draws—use the seed for reproducible aggregate distributions in load tests.

What validation errors occur with invalid weights?

Switchyard rejects configurations where any weight is non-finite (NaN, infinity) or negative, or where all weights are zero. Check server logs at startup for TOML validation failures.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}