How to Configure Weighted Random Routing in Switchyard for A/B Testing

Switchyard implements A/B testing through its random routing algorithm, allowing you to distribute traffic across target models by defining relative weights in a TOML configuration file or via the Python API.

Weighted random routing in Switchyard enables controlled traffic distribution for A/B testing scenarios without external load balancers. This feature, part of the NVIDIA-NeMo/Switchyard repository, processes requests by selecting from multiple backend targets according to configurable probability distributions defined in your route manifest.

Random Routing Algorithm Implementation

The core logic resides in crates/libsy/src/algorithms/rand.rs, which implements the random routing algorithm using Rust's WeightedIndex distribution. When a request hits a route where type = "random", Switchyard evaluates the weights array to calculate selection probabilities for each target in the targets list.

The algorithm supports deterministic routing through an optional seed parameter, ensuring reproducible target selection across server restarts—critical for debugging or controlled experiments.

TOML Configuration for Weighted Routes

Define weighted random routes in your TOML manifest (e.g., dev-server/config.toml) using the following schema:

[routes.ab_test]
id      = "ab_test"
type    = "random"
targets = ["capable", "efficient"]
weights = [1, 3]
seed    = 42

Key parameters:

• targets: Array of model identifiers defined elsewhere in your configuration
• weights: Optional array of relative probabilities (defaults to 1 for all targets if omitted)
• seed: Optional integer for deterministic selection

Weight Validation Rules

Switchyard validates weight arrays at server startup in crates/libsy/src/algorithms/rand.rs. The system enforces three constraints:

• The weights array length must exactly match the targets array length
• All values must be finite, non-negative numbers
• At least one weight must be greater than 0

Violations raise a SwitchyardError immediately at startup, preventing invalid routing configurations from serving traffic.

Python API for Dynamic Route Construction

The Python bindings in crates/switchyard-py/src/libsy_bindings.rs expose the random() function through PyO3, wrapped by switchyard/libsy/algorithms.py. Construct routes programmatically when building dynamic configurations:

from switchyard.libsy import algorithms

# Create weighted-random routing algorithm programmatically

ab_test_algo = algorithms.random(
    targets=["capable", "efficient"],
    weights=[1, 3],
    seed=42
)

This returns a routing algorithm instance that can be attached to Switchyard clients or passed to server configuration builders. The validation logic mirrors the TOML path, ensuring consistency across configuration methods.

CLI Usage and Client Integration

Once configured, clients invoke weighted routes via the --model flag:

switchyard run --model ab_test --prompt "Explain quantum entanglement in simple terms."

Switchyard selects the target according to the supplied weights (in the example above, "efficient" receives 75% of traffic while "capable" receives 25%) and forwards the request to the selected backend.

Summary

• Weighted random routing in Switchyard uses relative weights (not percentages) to distribute traffic across multiple targets
• Configuration occurs in TOML files through the [routes.<name>] block with type = "random"
• The implementation resides in crates/libsy/src/algorithms/rand.rs using WeightedIndex for selection
• Validation occurs at server startup, requiring matching array lengths and at least one positive weight
• Python API access is available through switchyard.libsy.algorithms.random()
• Optional seed parameters enable deterministic, reproducible routing decisions

Frequently Asked Questions

What is the difference between relative weights and percentages in Switchyard?

Switchyard interprets weights as relative probabilities rather than strict percentages. A configuration with weights = [1, 3] assigns 25% traffic to the first target and 75% to the second, but Switchyard calculates this proportionally rather than requiring values that sum to 100. If weights are omitted, every target receives an implicit weight of 1, resulting in uniform random distribution.

How does the seed parameter enable deterministic routing?

Providing a seed integer initializes the random number generator with a fixed state, causing Switchyard to select the same sequence of targets for identical request sequences across server restarts. This reproducibility, implemented in crates/libsy/src/algorithms/rand.rs, allows developers to debug routing behavior or maintain consistent experimental conditions during A/B testing.

What happens if I provide invalid weights in my TOML configuration?

Switchyard validates weight arrays immediately at server startup. If you provide negative numbers, non-finite values, a mismatched array length, or an array where all weights equal zero, the system raises a SwitchyardError and prevents the server from starting. This validation occurs in the Rust layer before any traffic processing begins, ensuring no invalid routing states reach production.

Can I use weighted random routing with more than two targets for multivariate testing?

Yes. The targets and weights arrays support any number of backend models. Simply extend both arrays equally—targets = ["model_a", "model_b", "model_c"] with weights = [2, 1, 1] sends 50% traffic to model_a and 25% to each of the others. The WeightedIndex implementation in crates/libsy/src/algorithms/rand.rs efficiently handles arbitrary target counts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}