How WeightPerturbationSigma Enables Per-User Score Randomization in X-Algorithm A/B Testing

WeightPerturbationSigma injects Gaussian noise with standard deviation σ into raw recommendation logits at the scoring stage, enabling controlled per-user randomization for A/B tests without modifying underlying model weights.

The xai-org/x-algorithm repository implements this perturbation strategy to facilitate statistical inference in large-scale recommendation systems. By applying noise at inference time rather than training time, the platform maintains model consistency while creating independently randomized treatment groups for experimentation.

Where Weight Perturbation Lives in the Codebase

The perturbation mechanism resides in the scoring pipeline rather than the model forward pass. In bdsm/runtime/score_results_sink_focal.py, the system intercepts raw logits produced by the two-tower retrieval model and applies the noise injection before final ranking. The underlying model in phoenix/xrex/models/recsys_two_tower_model.py generates deterministic score vectors, while phoenix/xrex/utils/model.py supplies the JAX random utilities required for reproducible noise generation.

This separation of concerns ensures that the core recommendation logic remains unchanged—only the final score presentation varies per user based on the σ parameter.

How σ Controls Per-User Randomization

The σ (sigma) hyper-parameter defines the standard deviation of a zero-mean Gaussian distribution. For each user request, the system draws a fresh random tensor ε ~ N(0, σ²) and adds it to the model’s raw logits:

# Conceptual implementation from score_results_sink_focal.py

rng = jax.random.PRNGKey(seed)
eps = jax.random.normal(rng, logits.shape) * weight_perturbation_sigma
perturbed_logits = logits + eps

• Small σ (e.g., 0.01) produces subtle variations suitable for detecting minor ranking changes.
• Large σ (e.g., 0.1) introduces significant randomness for stress-testing the pipeline’s robustness.

Because the noise is sampled independently for each user, the system achieves per-user randomization without requiring separate model shards or bucket-assignment logic.

Configuring WeightPerturbationSigma via Hydra

The platform exposes σ through Hydra-style YAML configurations, allowing data scientists to adjust randomization levels without code changes. The experiment flag propagates directly to the scoring sink:

# config/ab_test.yaml

experiment:
  weight_perturbation:
    enabled: true
    sigma: 0.05   # Standard deviation for Gaussian noise injection

When enabled is false or σ is zero, the pipeline bypasses perturbation entirely, ensuring backward compatibility with existing control groups.

Implementation in the Scoring Pipeline

The actual perturbation occurs within the apply_weight_perturbation function, which handles edge cases and maintains deterministic randomness across batches:

# score_results_sink_focal.py – excerpt

def apply_weight_perturbation(logits, sigma, rng):
    """Add Gaussian noise with std-dev σ to each user's logits."""
    if sigma <= 0.0:
        return logits                     # No-op path for control groups

    eps = jax.random.normal(rng, logits.shape) * sigma
    return logits + eps

# Pipeline integration

if cfg.experiment.weight_perturbation.enabled:
    rng = jax.random.PRNGKey(batch_id)   # Deterministic per-batch seed

    perturbed_logits = apply_weight_perturbation(
        raw_logits, 
        cfg.experiment.weight_perturbation.sigma, 
        rng
    )
else:
    perturbed_logits = raw_logits

Using batch_id as the random seed ensures that rerunning the same batch yields identical perturbations, critical for debugging and reproducibility.

Statistical Benefits for A/B Testing

Exchangeability preservation stands as the primary advantage of this approach. Because the Gaussian noise is independent across users and identically distributed, the randomization satisfies the exchangeability assumption required for unbiased treatment effect estimation.

The method also provides operational simplicity. Both control and variant arms execute identical inference code; only the σ parameter differs. This eliminates risks associated with model version skew or divergent serving paths that typically complicate A/B infrastructure.

Finally, because perturbation applies after logit generation but before the final ranking operation, core relevance metrics such as NDCG and Hit Rate remain comparable across arms. The noise primarily affects the relative ordering of near-tied items, which is precisely the variance A/B tests aim to measure.

Summary

• WeightPerturbationSigma is defined in the scoring configuration and controls the standard deviation of injected Gaussian noise.
• The implementation lives in bdsm/runtime/score_results_sink_focal.py, modifying logits produced by phoenix/xrex/models/recsys_two_tower_model.py.
• Noise injection uses JAX random utilities from phoenix/xrex/utils/model.py with deterministic seeds per batch.
• Per-user randomization is achieved by sampling independent noise tensors for each inference request.
• The approach preserves model consistency while enabling statistically valid A/B testing without separate serving paths.

Frequently Asked Questions

What is WeightPerturbationSigma in the X-Algorithm codebase?

WeightPerturbationSigma is a configuration parameter that specifies the standard deviation (σ) of Gaussian noise added to recommendation scores during the inference phase. It enables per-user randomization by perturbing raw logits before they are converted into final rankings, facilitating A/B testing without altering trained model weights.

How does per-user score randomization differ from model weight perturbation?

Per-user score randomization injects noise at the output layer (logits) for individual inference requests, leaving model weights intact. Model weight perturbation, by contrast, modifies the trained parameters themselves. The scoring-stage approach ensures that the same model serves both control and treatment groups, eliminating version skew and reducing infrastructure complexity.

Why is Gaussian noise specifically used for the perturbation?

Gaussian (normal) distribution noise is used because it provides zero-mean perturbations that preserve the expected value of the original scores while introducing controlled variance. The mathematical properties of the normal distribution—specifically the ability to tune variance via σ—make it ideal for maintaining statistical exchangeability across users, a requirement for valid causal inference in A/B testing.

How does the system ensure reproducibility when applying random noise?

The system uses deterministic random seeds derived from batch_id when calling jax.random.PRNGKey() in the scoring sink. This ensures that identical input batches generate identical perturbation patterns across repeated runs, enabling debugging and consistency verification while maintaining statistical independence across different users.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}