# How DeepEP Uses Zero‑Copy Optimization to Reduce SM Usage

> Discover how DeepEP s zero copy optimization slashes shared memory usage by reading directly from RDMA buffers, boosting GPU occupancy and kernel efficiency. Learn more today.

- Repository: [DeepSeek/DeepEP](https://github.com/deepseek-ai/DeepEP)
- Tags: deep-dive
- Published: 2026-04-25

---

**DeepEP eliminates intermediate shared‑memory copies by reading directly from pre‑registered RDMA buffers when the `zero_copy` flag is enabled, drastically lowering shared‑memory (SM) allocation per kernel and increasing GPU occupancy.**

DeepEP, developed by DeepSeek‑AI, accelerates expert‑parallel communication through low‑latency kernels that minimize memory overhead. When tensors already reside in the RDMA buffer used for the next combine step, the library can bypass the traditional staging buffer in shared memory. This zero‑copy optimization reduces SM usage and frees resources for compute‑intensive operations.

## Zero‑Copy Mechanism in the CUDA Kernels

The core zero‑copy logic resides in `csrc/kernels/internode_ll.cu`, where the combine kernel conditionally skips shared‑memory allocation based on the `zero_copy` parameter.

### The Combine Kernel Flag

At lines **740‑754**, the `combine` kernel receives a boolean `zero_copy` flag as part of its template parameters or kernel arguments. This flag determines whether the kernel should load data from the user‑provided tensor or reuse the existing RDMA buffer contents.

### Conditional Pointer Selection

Lines **850‑854** implement the critical branch: if `zero_copy` is `true`, the kernel sets the source pointer to `buf_ptr` (the pre‑registered RDMA buffer); otherwise, it reads from `x_int4` (the user tensor) and copies into shared memory. This single conditional eliminates the `memcpy` through SM when the data is already in the correct location.

### Constraints and Validation

The library enforces architectural constraints at lines **1166‑1184**. Specifically, `low_latency_combine` asserts that `zero_copy` cannot be combined with LogFMT compression, ensuring that incompatible optimization paths do not conflict.

## Python API and Buffer Management

The high‑level Python interface in [`deep_ep/buffer.py`](https://github.com/deepseek-ai/DeepEP/blob/main/deep_ep/buffer.py) exposes the zero‑copy capability while managing the underlying RDMA buffer lifecycle.

### Exposing the zero_copy Argument

Lines **618‑639** define the `low_latency_combine` method signature, which includes the `zero_copy` boolean. The documentation specifies that this flag should be used cooperatively with `get_next_low_latency_combine_buffer`, indicating that the caller must ensure the tensor already resides in the RDMA buffer before invoking zero‑copy mode.

### Dispatch Forwarding

At lines **658‑660**, the dispatch side forwards the `zero_copy` flag to the native C++ implementation, ensuring that the optimization propagates through the entire call stack.

### C++ Wrapper Layer

The [`csrc/deep_ep.cpp`](https://github.com/deepseek-ai/DeepEP/blob/main/csrc/deep_ep.cpp) file (lines **1684‑1772**) contains the pybind11 wrappers for `low_latency_dispatch` and `low_latency_combine`. These functions accept the `zero_copy` flag from Python and pass it directly to the CUDA kernel launch configuration, bridging the Python API and the device code.

## How Zero‑Copy Reduces SM Usage

Understanding the memory path reveals why zero‑copy decreases SM pressure.

**Standard Path (Copy‑In)**

In the default mode, the kernel first loads the user tensor `x` into a shared‑memory staging buffer (`smem_buffer`). This buffer must accommodate the full message payload—typically `hidden * sizeof(bfloat16)` plus metadata—consuming significant SM resources per thread block.

**Zero‑Copy Path**

When `zero_copy == true`, the kernel skips the load from `x`. Instead, it reads directly from the pre‑registered RDMA buffer (`buf_ptr`). Consequently, the shared‑memory allocation for the payload can be omitted or drastically reduced, retaining only metadata and synchronization structures.

Because modern GPUs limit SM to approximately 64 KB per streaming multiprocessor, freeing this buffer permits more thread blocks to be resident simultaneously. Higher occupancy directly translates to lower latency and improved throughput, especially when running many expert‑parallel ranks concurrently.

## End‑to‑End Implementation Example

The following workflow demonstrates how to utilize zero‑copy optimization in practice:

```python
import torch
import deep_ep

# Initialize buffer with low-latency mode enabled

buf = deep_ep.Buffer(
    rank=0,
    num_ranks=2,
    num_nvl_bytes=64 << 20,   # 64 MiB NVL buffer

    num_rdma_bytes=32 << 20,  # 32 MiB RDMA buffer

    low_latency_mode=True,
    explicitly_destroy=False,
    enable_shrink=False,
    use_fabric=False
)

# Standard dispatch: copies data into RDMA buffer

x = torch.randn([4, 128, 256], device='cuda', dtype=torch.bfloat16)
topk_idx = torch.randint(0, 256, (512, 4), device='cuda', dtype=deep_ep.topk_idx_t)

packed_recv_x, _, handle, event, _ = buf.low_latency_dispatch(
    x, topk_idx,
    use_fp8=False,
    round_scale=False,
    use_ue8m0=False,
    async_finish=False,
    return_recv_hook=False
)

# Retrieve the raw RDMA buffer for the next operation

next_buf = buf.get_next_low_latency_combine_buffer(handle)

# Combine with zero-copy: reads directly from RDMA buffer, skips SM staging

combined_x, combine_event, recv_hook = buf.low_latency_combine(
    next_buf,               # Already in RDMA buffer

    topk_idx,
    topk_weights=torch.ones_like(topk_idx, dtype=torch.float32),
    handle=handle,
    use_logfmt=False,
    zero_copy=True,         # Enable zero-copy optimization

    async_finish=False,
    return_recv_hook=False
)

print('Combined shape:', combined_x.shape)

```

In this flow, `get_next_low_latency_combine_buffer` provides direct access to the RDMA buffer that will hold the next combine's inputs. By setting `zero_copy=True`, the subsequent `low_latency_combine` call avoids allocating the large shared‑memory staging area, reducing SM usage as implemented in `csrc/kernels/internode_ll.cu`.

## Summary

- **Zero‑copy mode** in DeepEP allows kernels to read directly from RDMA buffers, bypassing shared‑memory staging entirely.
- The optimization is controlled by the `zero_copy` boolean flag passed through [`deep_ep/buffer.py`](https://github.com/deepseek-ai/DeepEP/blob/main/deep_ep/buffer.py) and [`csrc/deep_ep.cpp`](https://github.com/deepseek-ai/DeepEP/blob/main/csrc/deep_ep.cpp) to the CUDA kernels in `csrc/kernels/internode_ll.cu`.
- **SM reduction** occurs because the kernel omits the `hidden * sizeof(bfloat16)` buffer, retaining only metadata in shared memory and enabling higher GPU occupancy.
- **Preconditions** require that tensors already reside in the RDMA buffer (via `get_next_low_latency_combine_buffer`) and that LogFMT compression is disabled.
- This pattern is essential for low‑latency expert‑parallel communication where maximizing resident thread blocks directly improves throughput.

## Frequently Asked Questions

### What is the difference between zero_copy=True and zero_copy=False in DeepEP?

When `zero_copy=False`, the `combine` kernel copies the input tensor from user memory into a shared‑memory staging buffer before processing, consuming significant SM resources. When `zero_copy=True`, the kernel reads directly from the pre‑registered RDMA buffer (`buf_ptr`), eliminating the shared‑memory copy and reducing the per‑block SM footprint to metadata only.

### Why does zero‑copy optimization specifically reduce SM usage?

Standard DeepEP kernels allocate a shared‑memory buffer sized for the full hidden dimension (typically `hidden * sizeof(bfloat16)`) to stage incoming data. By reading directly from the RDMA buffer when `zero_copy` is enabled, the kernel avoids this allocation. Since each streaming multiprocessor has limited shared memory (typically 64‑100 KB), reducing per‑block consumption allows more blocks to run concurrently, increasing occupancy.

### Can zero_copy be used with LogFMT compression in DeepEP?

No. According to the assertion at lines **1166‑1184** in `csrc/kernels/internode_ll.cu`, the `low_latency_combine` kernel explicitly forbids combining `zero_copy=True` with LogFMT compression. These two optimization paths are mutually exclusive in the current implementation.

### How do I prepare a tensor for zero‑copy combine operations?

First, execute `low_latency_dispatch` with `zero_copy=False` to transmit data into the RDMA buffer. Then, retrieve the raw buffer pointer using `get_next_low_latency_combine_buffer` (as documented in [`deep_ep/buffer.py`](https://github.com/deepseek-ai/DeepEP/blob/main/deep_ep/buffer.py) lines **618‑639**). Finally, pass this buffer to `low_latency_combine` with `zero_copy=True`, ensuring the data is already in the RDMA memory space and avoiding the internal copy step.