Rejection Sampling with Residual Correction in MTPLX: Exact Sampling Explained

Rejection sampling with residual correction in MTPLX is an exact sampling algorithm that accepts draft model tokens when they align with the target distribution, and otherwise samples from the residual probability mass to guarantee statistically identical outputs to the target model.

MTPLX is an open-source inference acceleration framework that implements speculative decoding to reduce latency in transformer inference. The repository uses rejection sampling with residual correction to combine the speed of a lightweight draft model with the statistical fidelity of the full target model, ensuring no approximation errors are introduced during the acceleration process.

How Rejection Sampling with Residual Correction Works

The algorithm operates through three distinct phases that ensure exact compliance with the target model’s probability distribution.

Draft Generation Phase

The process begins with a draft model proposing candidate tokens. For a given token position, the draft model outputs a probability distribution draft_q over the vocabulary. A token is sampled from this distribution and proposed as the speculative output. This step is computationally cheap because the draft model is significantly smaller than the target model.

The Exact Rejection Test

MTPLX performs a statistical verification using a uniform random variable. A value u is drawn from the uniform distribution [0, 1). The draft token is accepted if:

u < draft_q(token)

If this condition is met, the token is emitted immediately without invoking the target model. This acceptance criterion ensures that tokens are only kept when the draft model’s probability mass supports their validity under the target distribution.

Residual Distribution Correction

When the rejection test fails (u >= draft_q(token)), MTPLX executes residual correction to sample from the uncovered probability mass. The system computes the residual distribution as the element-wise difference between the target probability target_p and the draft probability draft_q, clipped to non-negative values:

residual = np.maximum(target_p - draft_q, 0.0)

This residual mass is then renormalized to form a valid probability distribution. A new token is sampled from this residual distribution, ensuring that the overall sampling probability equals the original target distribution exactly. According to the MTPLX source code, this correction guarantees that the final output follows the target model’s categorical distribution without bias.

Implementation in the MTPLX Source Code

The rejection sampling mechanism is distributed across several key components in the repository:

• mtplx/sampling.py (line 252): Contains the residual_distribution function that implements the subtraction, clipping, and renormalization logic required to compute the residual mass.

• mtplx/backends/hy_v3_mtp.py (line 61): Includes the high-level comment and logic describing the exact rejection sampling verification step that determines whether to accept the draft token or trigger residual correction.

• mtplx/thinking_guard.py (line 23): Integrates the residual correction into the model’s execution loop through the thinking_guard and step3p5_mtp_patch functions, ensuring seamless propagation of corrected tokens through the runtime.

• tests/test_sampling.py (line 12): Validates the residual-correction path through unit tests that ensure rejected tokens emit the residual distribution and that corrected tokens follow the target distribution statistics.

• tests/test_snapshot_free_rejection_repair.py: Contains end-to-end tests verifying the repair mechanism for rejected tokens in production inference scenarios.

Practical Implementation Example

The following example demonstrates the core rejection sampling logic using the MTPLX API. This pattern mirrors the implementation found in mtplx/sampling.py:

import numpy as np
from mtplx.sampling import residual_distribution

def reject_sampling_with_residual(target_logits, draft_logits):
    """
    Perform exact rejection sampling with residual correction.
    Returns the sampled token id.
    """
    # Convert logits to probabilities

    target_p = np.exp(target_logits) / np.exp(target_logits).sum()
    draft_q = np.exp(draft_logits) / np.exp(draft_logits).sum()

    # Sample a draft token

    draft_token = np.random.choice(len(draft_q), p=draft_q)

    # Rejection test

    u = np.random.rand()
    if u < draft_q[draft_token]:
        # Accept draft token

        return draft_token
    else:
        # Compute residual distribution and sample from it

        residual = residual_distribution(target_p, draft_q)
        return np.random.choice(residual.token_ids, p=residual.probs)

# Example usage

target = np.random.randn(50257)   # pretend target logits

draft  = target * 0.8 + np.random.randn(50257) * 0.2  # noisy draft logits

sampled_token = reject_sampling_with_residual(target, draft)
print(f"Sampled token id: {sampled_token}")

In this implementation, residual_distribution handles the computation of np.maximum(target_p - draft_q, 0.0) and subsequent normalization. The function first attempts to accept the draft token; upon rejection, it falls back to the residual distribution, maintaining exact statistical equivalence with the target model.

Why Exact Rejection Sampling Matters

MTPLX employs exact rejection sampling rather than approximate heuristic methods to achieve three critical objectives:

• Statistical Correctness: Because the residual correction step accounts for the entire probability mass that the draft did not cover, the combined procedure reproduces the exact categorical distribution of the target model. No bias is introduced, unlike top-k truncation or temperature scaling approximations.

• Inference Speed: The draft path is computationally inexpensive; when tokens are accepted, the system avoids expensive target model forward passes. Residual correction only occurs when necessary, minimizing overhead.

• Multi-Token Compatibility: The same residual machinery works for both single-token and multi-token speculative (MTP) drafts, as the residual distribution can be constructed for arbitrary draft shapes and sequence lengths.

Summary

• Rejection sampling with residual correction in MTPLX guarantees exact target distribution compliance by accepting draft tokens when u < draft_q and otherwise sampling from the residual mass.
• The residual distribution is computed as np.maximum(target_p - draft_q, 0.0) and renormalized to ensure valid probability distributions.
• Key implementation files include mtplx/sampling.py for the calculation logic, mtplx/backends/hy_v3_mtp.py for the rejection test, and mtplx/thinking_guard.py for runtime integration.
• This approach delivers identical output statistics to target-only decoding while achieving substantial speedups through speculative execution.

Frequently Asked Questions

What distinguishes standard rejection sampling from residual correction in MTPLX?

Standard rejection sampling typically discards rejected samples without recovery. In MTPLX, residual correction transforms the rejection into a targeted sampling operation from the residual distribution—the precise probability mass where the draft model underrepresented the target. This ensures that every sampling step produces a valid token from the exact target distribution rather than requiring a full restart.

Where is the residual distribution calculated in the MTPLX codebase?

The residual distribution calculation is implemented in mtplx/sampling.py at line 252 within the residual_distribution function. This utility performs the element-wise subtraction of draft probabilities from target probabilities, applies the non-negative clip operation, and renormalizes the result to produce a valid sampling distribution.

Does residual correction introduce any approximation errors?

No. Residual correction in MTPLX is mathematically exact. By construction, the sum of the accepted draft mass and the residual mass equals the full target probability distribution. The algorithm guarantees that the final token distribution matches what would be produced by sampling directly from the target model, eliminating the approximation errors common in speculative decoding heuristics.

How does rejection sampling with residual correction improve inference speed?

The mechanism accelerates inference by short-circuiting target model evaluation for tokens where the cheap draft model produces statistically acceptable predictions. When the uniform random sample falls below the draft probability, the token is accepted immediately. Only rejected tokens trigger the residual correction path, which requires target model evaluation but occurs significantly less frequently than standard autoregressive decoding steps.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}