How DeepEP Uses Zero‑Copy Optimization to Reduce SM Usage

DeepEP eliminates intermediate shared‑memory copies by reading directly from pre‑registered RDMA buffers when the zero_copy flag is enabled, drastically lowering shared‑memory (SM) allocation per kernel and increasing GPU occupancy.

DeepEP, developed by DeepSeek‑AI, accelerates expert‑parallel communication through low‑latency kernels that minimize memory overhead. When tensors already reside in the RDMA buffer used for the next combine step, the library can bypass the traditional staging buffer in shared memory. This zero‑copy optimization reduces SM usage and frees resources for compute‑intensive operations.

Zero‑Copy Mechanism in the CUDA Kernels

The core zero‑copy logic resides in csrc/kernels/internode_ll.cu, where the combine kernel conditionally skips shared‑memory allocation based on the zero_copy parameter.

The Combine Kernel Flag

At lines 740‑754, the combine kernel receives a boolean zero_copy flag as part of its template parameters or kernel arguments. This flag determines whether the kernel should load data from the user‑provided tensor or reuse the existing RDMA buffer contents.

Conditional Pointer Selection

Lines 850‑854 implement the critical branch: if zero_copy is true, the kernel sets the source pointer to buf_ptr (the pre‑registered RDMA buffer); otherwise, it reads from x_int4 (the user tensor) and copies into shared memory. This single conditional eliminates the memcpy through SM when the data is already in the correct location.

Constraints and Validation

The library enforces architectural constraints at lines 1166‑1184. Specifically, low_latency_combine asserts that zero_copy cannot be combined with LogFMT compression, ensuring that incompatible optimization paths do not conflict.

Python API and Buffer Management

The high‑level Python interface in deep_ep/buffer.py exposes the zero‑copy capability while managing the underlying RDMA buffer lifecycle.

Exposing the zero_copy Argument

Lines 618‑639 define the low_latency_combine method signature, which includes the zero_copy boolean. The documentation specifies that this flag should be used cooperatively with get_next_low_latency_combine_buffer, indicating that the caller must ensure the tensor already resides in the RDMA buffer before invoking zero‑copy mode.

Dispatch Forwarding

At lines 658‑660, the dispatch side forwards the zero_copy flag to the native C++ implementation, ensuring that the optimization propagates through the entire call stack.

C++ Wrapper Layer

The csrc/deep_ep.cpp file (lines 1684‑1772) contains the pybind11 wrappers for low_latency_dispatch and low_latency_combine. These functions accept the zero_copy flag from Python and pass it directly to the CUDA kernel launch configuration, bridging the Python API and the device code.

How Zero‑Copy Reduces SM Usage

Understanding the memory path reveals why zero‑copy decreases SM pressure.

Standard Path (Copy‑In)

In the default mode, the kernel first loads the user tensor x into a shared‑memory staging buffer (smem_buffer). This buffer must accommodate the full message payload—typically hidden * sizeof(bfloat16) plus metadata—consuming significant SM resources per thread block.

Zero‑Copy Path

When zero_copy == true, the kernel skips the load from x. Instead, it reads directly from the pre‑registered RDMA buffer (buf_ptr). Consequently, the shared‑memory allocation for the payload can be omitted or drastically reduced, retaining only metadata and synchronization structures.

Because modern GPUs limit SM to approximately 64 KB per streaming multiprocessor, freeing this buffer permits more thread blocks to be resident simultaneously. Higher occupancy directly translates to lower latency and improved throughput, especially when running many expert‑parallel ranks concurrently.

End‑to‑End Implementation Example

The following workflow demonstrates how to utilize zero‑copy optimization in practice:

import torch
import deep_ep

# Initialize buffer with low-latency mode enabled

buf = deep_ep.Buffer(
    rank=0,
    num_ranks=2,
    num_nvl_bytes=64 << 20,   # 64 MiB NVL buffer

    num_rdma_bytes=32 << 20,  # 32 MiB RDMA buffer

    low_latency_mode=True,
    explicitly_destroy=False,
    enable_shrink=False,
    use_fabric=False
)

# Standard dispatch: copies data into RDMA buffer

x = torch.randn([4, 128, 256], device='cuda', dtype=torch.bfloat16)
topk_idx = torch.randint(0, 256, (512, 4), device='cuda', dtype=deep_ep.topk_idx_t)

packed_recv_x, _, handle, event, _ = buf.low_latency_dispatch(
    x, topk_idx,
    use_fp8=False,
    round_scale=False,
    use_ue8m0=False,
    async_finish=False,
    return_recv_hook=False
)

# Retrieve the raw RDMA buffer for the next operation

next_buf = buf.get_next_low_latency_combine_buffer(handle)

# Combine with zero-copy: reads directly from RDMA buffer, skips SM staging

combined_x, combine_event, recv_hook = buf.low_latency_combine(
    next_buf,               # Already in RDMA buffer

    topk_idx,
    topk_weights=torch.ones_like(topk_idx, dtype=torch.float32),
    handle=handle,
    use_logfmt=False,
    zero_copy=True,         # Enable zero-copy optimization

    async_finish=False,
    return_recv_hook=False
)

print('Combined shape:', combined_x.shape)

In this flow, get_next_low_latency_combine_buffer provides direct access to the RDMA buffer that will hold the next combine's inputs. By setting zero_copy=True, the subsequent low_latency_combine call avoids allocating the large shared‑memory staging area, reducing SM usage as implemented in csrc/kernels/internode_ll.cu.

Summary

• Zero‑copy mode in DeepEP allows kernels to read directly from RDMA buffers, bypassing shared‑memory staging entirely.
• The optimization is controlled by the zero_copy boolean flag passed through deep_ep/buffer.py and csrc/deep_ep.cpp to the CUDA kernels in csrc/kernels/internode_ll.cu.
• SM reduction occurs because the kernel omits the hidden * sizeof(bfloat16) buffer, retaining only metadata in shared memory and enabling higher GPU occupancy.
• Preconditions require that tensors already reside in the RDMA buffer (via get_next_low_latency_combine_buffer) and that LogFMT compression is disabled.
• This pattern is essential for low‑latency expert‑parallel communication where maximizing resident thread blocks directly improves throughput.

Frequently Asked Questions

What is the difference between zero_copy=True and zero_copy=False in DeepEP?

When zero_copy=False, the combine kernel copies the input tensor from user memory into a shared‑memory staging buffer before processing, consuming significant SM resources. When zero_copy=True, the kernel reads directly from the pre‑registered RDMA buffer (buf_ptr), eliminating the shared‑memory copy and reducing the per‑block SM footprint to metadata only.

Why does zero‑copy optimization specifically reduce SM usage?

Standard DeepEP kernels allocate a shared‑memory buffer sized for the full hidden dimension (typically hidden * sizeof(bfloat16)) to stage incoming data. By reading directly from the RDMA buffer when zero_copy is enabled, the kernel avoids this allocation. Since each streaming multiprocessor has limited shared memory (typically 64‑100 KB), reducing per‑block consumption allows more blocks to run concurrently, increasing occupancy.

Can zero_copy be used with LogFMT compression in DeepEP?

No. According to the assertion at lines 1166‑1184 in csrc/kernels/internode_ll.cu, the low_latency_combine kernel explicitly forbids combining zero_copy=True with LogFMT compression. These two optimization paths are mutually exclusive in the current implementation.

How do I prepare a tensor for zero‑copy combine operations?

First, execute low_latency_dispatch with zero_copy=False to transmit data into the RDMA buffer. Then, retrieve the raw buffer pointer using get_next_low_latency_combine_buffer (as documented in deep_ep/buffer.py lines 618‑639). Finally, pass this buffer to low_latency_combine with zero_copy=True, ensuring the data is already in the RDMA memory space and avoiding the internal copy step.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}