DeepEP FP8 Dispatch and Combine Operations: Low-Precision All-to-All for MoE

DeepEP implements FP8 dispatch and combine operations as a two-stage process that quantizes BF16 activations to FP8-E4M3 with per-128-channel scaling before transmission, then de-quantizes them back to BF16 during the combine reduction, reducing inter-GPU communication bandwidth by approximately 4× while preserving numerical accuracy through floating-point scale factors.

DeepEP (Deep Expert Parallelism) is an open-source communication library by DeepSeek-AI that accelerates Mixture-of-Experts (MoE) training and inference. The DeepEP FP8 dispatch and combine operations provide an optimized low-precision path for all-to-all communication, cutting message sizes by nearly 75% through hardware-accelerated FP8 quantization and custom CUDA kernels.

How FP8 Dispatch and Combine Operations Work

The FP8 path consists of two distinct stages: dispatch (quantization and send) and combine (receive and de-quantization). This architecture leverages the internode_ll.cu kernels to minimize RDMA/NVLink traffic without sacrificing the dynamic range of the original BF16 activations.

The Dispatch Stage: Quantization and Scale Computation

During the dispatch phase, the kernel processes BF16 activations by dividing the hidden dimension into 128-element blocks. For each block, it computes the maximum absolute value (amax) and derives a per-block scale using scale = 127 / amax. This logic is implemented in csrc/kernels/utils.cuh within the calculate_fp8_scales function.

The actual quantization uses the NVIDIA intrinsic __nv_cvt_float2_to_fp8x2 to convert BF16 values to FP8-E4M3 format. The kernel then packs both the quantized data (as int2 types) and the floating-point scales into the message buffer. In csrc/kernels/internode_ll.cu, this logic resides inside the dispatch template specialization where if constexpr (kUseFP8) evaluates to true (approximately lines 220-250).

Message Transmission via IBGDA

After packing, the kernel transmits messages via IBGDA (InfiniBand GPUDirect Async), which combines NVSHMEM with RDMA for zero-copy transfers. Each message contains the FP8 payload followed by the scale factors, adding only approximately 0.78% overhead (one float per 128 elements).

The FP8 Combine Operation

The combine stage reverses the quantization process. The receiving GPU extracts the FP8 payload and per-block scales from the incoming message buffer using the rdma_x_scales pointer. Using the inverse scale (scale_inv) stored in the message, the kernel de-quantizes values back to full-precision BF16 by multiplying the FP8 values by scale_inv and casting through FP32.

This reduction happens in the combine template in csrc/kernels/internode_ll.cu (approximately line 440), where the kUseFP8 specialization accumulates de-quantized values into the combined_x buffer. If top-k weights are provided, they undergo the same reduction process alongside the activations.

Python API for FP8 Operations

The deep_ep.Buffer class exposes these kernels through high-level Python methods. To enable FP8 quantization during dispatch:

recv_x, recv_idx, recv_weights, recv_counts, handle, ev = \
    buffer.dispatch(
        x,                     # BF16 input tensor

        topk_idx=topk_idx,
        topk_weights=topk_weights,
        use_fp8=True,         # Enable FP8 quantization path

        round_scale=False,    # Optional: round scale to power-of-2

        use_ue8m0=False)      # Optional: use packed UE8M0 format

The returned handle stores layout metadata required for the matching combine operation. To de-quantize and reduce during combine:

combined_x, combined_weights, ev = buffer.combine(
    recv_x,                # Target buffer (BF16)

    handle,                # Layout from dispatch

    topk_weights=recv_weights,
    use_fp8=True)          # Expect FP8 payload and apply de-quantization

Advanced Configuration Options

DeepEP provides two optional modes that modify how scale factors are handled:

• Round-Scale Mode: When round_scale=True, the kernel rounds the computed scale to the nearest power-of-two in calculate_fp8_scales. This improves reproducibility across GPUs by ensuring the scale factor has an exact binary representation in FP8 hardware.

• UE8M0 Support: Setting use_ue8m0=True activates an experimental path handled by the kUseUE8M0 template parameter. This mode stores scale factors in a packed 8-bit format rather than full 32-bit floats, further reducing metadata overhead.

The FP8 path reduces communication bandwidth by approximately 4× compared to BF16, making it essential for large-scale MoE models where all-to-all communication dominates execution time.

Summary

• DeepEP FP8 dispatch quantizes BF16 activations to FP8-E4M3 using per-128-channel scales calculated as 127 / amax in csrc/kernels/internode_ll.cu.
• The combine operation de-quantizes received FP8 data by multiplying with inverse scales stored in the message, accumulating results in BF16 precision.
• The Python API exposes these operations through buffer.dispatch(use_fp8=True) and buffer.combine(use_fp8=True) in deep_ep/buffer.py.
• Per-128-channel scaling prevents overflow in FP8-E4M3’s limited 4-bit exponent range while adding minimal overhead (~0.78%).
• Optional round_scale and use_ue8m0 modes provide hardware-compatible scaling factors and reduced metadata overhead respectively.

Frequently Asked Questions

Why does DeepEP use per-128-channel scaling instead of a global scale?

FP8-E4M3 provides only 4 exponent bits, which cannot represent the dynamic range of typical transformer hidden dimensions (e.g., 7168) with a single global scale. Per-128-channel scaling creates 128-element chunks with independent scale factors, preventing overflow while keeping metadata overhead minimal at approximately one float per 128 elements.

How do the round_scale and use_ue8m0 parameters affect FP8 operations?

When round_scale=True, the kernel rounds scales to the nearest power-of-two in calculate_fp8_scales, ensuring exact hardware representation across different GPU architectures. The use_ue8m0 parameter activates an experimental packed 8-bit scale format handled by the kUseUE8M0 template parameter, reducing scale metadata from 32 bits to 8 bits per channel block.

What is the expected bandwidth reduction when using DeepEP FP8 operations?

The FP8 path reduces communication bandwidth by approximately 4× compared to BF16, since FP8-E4M3 uses 8 bits per element versus 16 bits for BF16. The additional overhead from transmitting scale factors (roughly 0.78%) is negligible, making the effective compression ratio nearly 2:1 for the data payload alone and 4:1 when accounting for protocol overhead in practice.

Where is the FP8 quantization logic implemented in the DeepEP source code?

The quantization logic resides in csrc/kernels/internode_ll.cu within the dispatch template (lines 220-250), which calls calculate_fp8_scales from csrc/kernels/utils.cuh. The de-quantization occurs in the same file’s combine template around line 440, where the kernel multiplies FP8 values by scale_inv and accumulates into combined_x.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}