# DeepEP FP8 Dispatch and Combine Operations: Low-Precision All-to-All for MoE

> Discover DeepEP's FP8 dispatch and combine operations for efficient all-to-all communication in MoE models. Reduce bandwidth by 4x with minimal accuracy loss.

- Repository: [DeepSeek/DeepEP](https://github.com/deepseek-ai/DeepEP)
- Tags: deep-dive
- Published: 2026-04-25

---

**DeepEP implements FP8 dispatch and combine operations as a two-stage process that quantizes BF16 activations to FP8-E4M3 with per-128-channel scaling before transmission, then de-quantizes them back to BF16 during the combine reduction, reducing inter-GPU communication bandwidth by approximately 4× while preserving numerical accuracy through floating-point scale factors.**

DeepEP (Deep Expert Parallelism) is an open-source communication library by DeepSeek-AI that accelerates Mixture-of-Experts (MoE) training and inference. The **DeepEP FP8 dispatch and combine operations** provide an optimized low-precision path for all-to-all communication, cutting message sizes by nearly 75% through hardware-accelerated FP8 quantization and custom CUDA kernels.

## How FP8 Dispatch and Combine Operations Work

The FP8 path consists of two distinct stages: dispatch (quantization and send) and combine (receive and de-quantization). This architecture leverages the `internode_ll.cu` kernels to minimize RDMA/NVLink traffic without sacrificing the dynamic range of the original BF16 activations.

### The Dispatch Stage: Quantization and Scale Computation

During the dispatch phase, the kernel processes BF16 activations by dividing the hidden dimension into 128-element blocks. For each block, it computes the maximum absolute value (`amax`) and derives a per-block scale using `scale = 127 / amax`. This logic is implemented in `csrc/kernels/utils.cuh` within the `calculate_fp8_scales` function.

The actual quantization uses the NVIDIA intrinsic `__nv_cvt_float2_to_fp8x2` to convert BF16 values to FP8-E4M3 format. The kernel then packs both the quantized data (as `int2` types) and the floating-point scales into the message buffer. In `csrc/kernels/internode_ll.cu`, this logic resides inside the `dispatch` template specialization where `if constexpr (kUseFP8)` evaluates to true (approximately lines 220-250).

### Message Transmission via IBGDA

After packing, the kernel transmits messages via IBGDA (InfiniBand GPUDirect Async), which combines NVSHMEM with RDMA for zero-copy transfers. Each message contains the FP8 payload followed by the scale factors, adding only approximately 0.78% overhead (one float per 128 elements).

## The FP8 Combine Operation

The combine stage reverses the quantization process. The receiving GPU extracts the FP8 payload and per-block scales from the incoming message buffer using the `rdma_x_scales` pointer. Using the inverse scale (`scale_inv`) stored in the message, the kernel de-quantizes values back to full-precision BF16 by multiplying the FP8 values by `scale_inv` and casting through FP32.

This reduction happens in the `combine` template in `csrc/kernels/internode_ll.cu` (approximately line 440), where the `kUseFP8` specialization accumulates de-quantized values into the `combined_x` buffer. If top-k weights are provided, they undergo the same reduction process alongside the activations.

## Python API for FP8 Operations

The `deep_ep.Buffer` class exposes these kernels through high-level Python methods. To enable FP8 quantization during dispatch:

```python
recv_x, recv_idx, recv_weights, recv_counts, handle, ev = \
    buffer.dispatch(
        x,                     # BF16 input tensor

        topk_idx=topk_idx,
        topk_weights=topk_weights,
        use_fp8=True,         # Enable FP8 quantization path

        round_scale=False,    # Optional: round scale to power-of-2

        use_ue8m0=False)      # Optional: use packed UE8M0 format

```

The returned `handle` stores layout metadata required for the matching combine operation. To de-quantize and reduce during combine:

```python
combined_x, combined_weights, ev = buffer.combine(
    recv_x,                # Target buffer (BF16)

    handle,                # Layout from dispatch

    topk_weights=recv_weights,
    use_fp8=True)          # Expect FP8 payload and apply de-quantization

```

## Advanced Configuration Options

DeepEP provides two optional modes that modify how scale factors are handled:

- **Round-Scale Mode**: When `round_scale=True`, the kernel rounds the computed scale to the nearest power-of-two in `calculate_fp8_scales`. This improves reproducibility across GPUs by ensuring the scale factor has an exact binary representation in FP8 hardware.

- **UE8M0 Support**: Setting `use_ue8m0=True` activates an experimental path handled by the `kUseUE8M0` template parameter. This mode stores scale factors in a packed 8-bit format rather than full 32-bit floats, further reducing metadata overhead.

The FP8 path reduces communication bandwidth by approximately 4× compared to BF16, making it essential for large-scale MoE models where all-to-all communication dominates execution time.

## Summary

- **DeepEP FP8 dispatch** quantizes BF16 activations to FP8-E4M3 using per-128-channel scales calculated as `127 / amax` in `csrc/kernels/internode_ll.cu`.
- The **combine operation** de-quantizes received FP8 data by multiplying with inverse scales stored in the message, accumulating results in BF16 precision.
- The Python API exposes these operations through `buffer.dispatch(use_fp8=True)` and `buffer.combine(use_fp8=True)` in [`deep_ep/buffer.py`](https://github.com/deepseek-ai/DeepEP/blob/main/deep_ep/buffer.py).
- Per-128-channel scaling prevents overflow in FP8-E4M3’s limited 4-bit exponent range while adding minimal overhead (~0.78%).
- Optional `round_scale` and `use_ue8m0` modes provide hardware-compatible scaling factors and reduced metadata overhead respectively.

## Frequently Asked Questions

### Why does DeepEP use per-128-channel scaling instead of a global scale?

FP8-E4M3 provides only 4 exponent bits, which cannot represent the dynamic range of typical transformer hidden dimensions (e.g., 7168) with a single global scale. Per-128-channel scaling creates 128-element chunks with independent scale factors, preventing overflow while keeping metadata overhead minimal at approximately one float per 128 elements.

### How do the round_scale and use_ue8m0 parameters affect FP8 operations?

When `round_scale=True`, the kernel rounds scales to the nearest power-of-two in `calculate_fp8_scales`, ensuring exact hardware representation across different GPU architectures. The `use_ue8m0` parameter activates an experimental packed 8-bit scale format handled by the `kUseUE8M0` template parameter, reducing scale metadata from 32 bits to 8 bits per channel block.

### What is the expected bandwidth reduction when using DeepEP FP8 operations?

The FP8 path reduces communication bandwidth by approximately 4× compared to BF16, since FP8-E4M3 uses 8 bits per element versus 16 bits for BF16. The additional overhead from transmitting scale factors (roughly 0.78%) is negligible, making the effective compression ratio nearly 2:1 for the data payload alone and 4:1 when accounting for protocol overhead in practice.

### Where is the FP8 quantization logic implemented in the DeepEP source code?

The quantization logic resides in `csrc/kernels/internode_ll.cu` within the `dispatch` template (lines 220-250), which calls `calculate_fp8_scales` from `csrc/kernels/utils.cuh`. The de-quantization occurs in the same file’s `combine` template around line 440, where the kernel multiplies FP8 values by `scale_inv` and accumulates into `combined_x`.