# How DeepEP Manages MoE Expert Parallelism Communication in DeepSeek

> Discover how DeepEP manages MoE expert parallelism communication using automatic NVLink and RDMA routing for efficient, high-throughput training and low-latency inference.

- Repository: [DeepSeek/DeepEP](https://github.com/deepseek-ai/DeepEP)
- Tags: deep-dive
- Published: 2026-04-25

---

**DeepEP manages MoE expert parallelism communication through the `Buffer` class, which automatically routes traffic over NVLink for intra-node and RDMA for inter-node operations, wrapping high-performance C++ kernels for both high-throughput training and low-latency inference scenarios.**

DeepEP is DeepSeek's open-source communication library designed specifically for Mixture-of-Experts (MoE) models, addressing the complex all-to-all communication patterns required when tokens must be routed to different expert GPUs across a distributed cluster. This article examines how the library abstracts hardware-specific transports behind a unified Python API while maintaining the performance characteristics necessary for large-scale training and serving.

## Buffer Initialization and Topology Validation

The **`Buffer`** class serves as the central orchestrator for MoE expert parallelism communication, managing the lifecycle of communication buffers and hardware topology verification.

### NVLink Connectivity Checks

When constructing a `Buffer` instance, DeepEP first validates the physical interconnectivity between GPUs. The initialization calls **`check_nvlink_connections`** (located in [`deep_ep/utils.py`](https://github.com/deepseek-ai/DeepEP/blob/main/deep_ep/utils.py), lines 64-99), which queries NVML to ensure every GPU pair can communicate over NVLink. This validation is critical because the high-throughput intranode kernels require direct NVLink connectivity between participating devices.

### Runtime Creation and Synchronization

After topology validation, the constructor instantiates the underlying C++ runtime (`deep_ep_cpp.Buffer`) with specified buffer sizes for NVLink and RDMA transports ([`deep_ep/buffer.py`](https://github.com/deepseek-ai/DeepEP/blob/main/deep_ep/buffer.py), lines 68-94). The initialization then synchronizes device IDs, IPC handles, and NVSHMEM identifiers across the process group (lines 95-133), ensuring each rank can address shared buffers. Environment variables such as `NVSHMEM_DISABLE_P2P` and `NVSHMEM_IB_ENABLE_IBGDA` control NVSHMEM behavior during this phase.

## Communication Paths and Routing Logic

DeepEP implements three distinct communication paths, with automatic selection based on runtime configuration and hardware topology:

- **High-throughput intranode**: Uses NVLink via `intranode_dispatch` and `intranode_combine` kernels when `low_latency_mode=False` and all ranks reside on the same node.
- **High-throughput internode**: Combines RDMA (IBGDA) for cross-node traffic with NVLink for intra-node traffic when `low_latency_mode=False` and ranks span multiple nodes.
- **Low-latency all-to-all**: Employs RDMA-only (IBGDA) with specially tuned kernels when `low_latency_mode=True`, optionally allowing NVLink assistance via `allow_nvlink_for_low_latency_mode`.

The routing decisions occur within the **`dispatch`** and **`combine`** methods. The `dispatch` method checks `self.runtime.get_num_rdma_ranks()`; if the result exceeds 1, it forwards to `internode_dispatch`, otherwise it invokes the intranode kernel ([`deep_ep/buffer.py`](https://github.com/deepseek-ai/DeepEP/blob/main/deep_ep/buffer.py), lines 73-80). The `combine` method mirrors this logic (lines 34-40).

## Dispatch and Combine APIs

DeepEP exposes two primary communication primitives that form the backbone of MoE expert parallelism workflows.

### High-Throughput Dispatch and Combine

The **`dispatch`** method sends token tensors to the ranks owning the selected experts. It accepts the input tensor `x` and `topk_idx` indices, returning received tensors, layout metadata (`recv_counts`), and a **handle** for the subsequent combine operation. The **`combine`** method performs the reverse operation, reducing tensors back to origin ranks using the handle from dispatch and optional `topk_weights` for weighted aggregation.

### Low-Latency Mode

For latency-critical serving workloads, **`low_latency_dispatch`** and **`low_latency_combine`** provide optimized paths using IBGDA with optional FP8 compression. This mode requires explicit buffer zeroing before each call via **`clean_low_latency_buffer**`, and enforces a minimum QP depth (`self.nvshmem_qp_depth`) derived from environment variables ([`deep_ep/buffer.py`](https://github.com/deepseek-ai/DeepEP/blob/main/deep_ep/buffer.py), lines 102-110, 162-166).

## Implementation Structure and Configuration

The library organizes functionality across specific source files with clear separation of concerns:

| File | Responsibility |
|------|----------------|
| [`deep_ep/buffer.py`](https://github.com/deepseek-ai/DeepEP/blob/main/deep_ep/buffer.py) | Core `Buffer` class, runtime orchestration, dispatch/combine implementations, and configuration helpers (`get_dispatch_config`, `get_combine_config`, `set_num_sms`) |
| [`deep_ep/utils.py`](https://github.com/deepseek-ai/DeepEP/blob/main/deep_ep/utils.py) | Topology validation (`check_nvlink_connections`) and `EventOverlap` wrapper utilities |
| `deep_ep_cpp` | Low-level C++ extension containing the actual all-to-all kernels for NVLink/RDMA and IBGDA paths |

Configuration selection occurs through **`get_dispatch_config`** and **`get_combine_config`**, which return tuned `Config` objects specifying SM counts and thread block sizes based on the number of EP ranks (lines 322-352). Users can modify SM allocation via **`set_num_sms`** for workload-specific tuning.

## Practical Usage Example

The following example demonstrates the complete workflow for MoE expert parallelism communication:

```python
import torch
import torch.distributed as dist
from deep_ep import Buffer

# Initialize distributed group

dist.init_process_group(backend="nccl")
group = dist.group.WORLD

# Create EP buffer with 32 MiB NVLink and 64 MiB RDMA allocation

buf = Buffer(group, num_nvl_bytes=32 << 20, num_rdma_bytes=64 << 20)

# Prepare input tokens and routing indices

x = torch.randn(2048, 4096, dtype=torch.bfloat16, device='cuda')
topk_idx = torch.randint(0, 64, (2048, 2), device='cuda', dtype=torch.int64)

# Dispatch tokens to expert ranks

recv_x, recv_topk_idx, recv_topk_weights, recv_counts, handle, event = \
    buf.dispatch(x, topk_idx=topk_idx)

# Local expert computation (placeholder)

local_out = recv_x * 1.23

# Combine results back to origin ranks

final_x, final_weights, combine_event = \
    buf.combine(local_out, handle, topk_weights=recv_topk_weights)

# Low-latency path for serving scenarios

buf.clean_low_latency_buffer(num_max_dispatch_tokens_per_rank=256,
                             hidden=x.size(1),
                             num_experts=64)

lat_recv_x, lat_recv_cnt, lat_handle, lat_event, _ = \
    buf.low_latency_dispatch(x, topk_idx, num_max_dispatch_tokens_per_rank=256,
                             num_experts=64)

# After processing...

lat_out, lat_event, _ = \
    buf.low_latency_combine(lat_recv_x, topk_idx, topk_weights=None,
                            handle=lat_handle)

```

## Summary

- **DeepEP** centralizes MoE expert parallelism communication through the `Buffer` class in [`deep_ep/buffer.py`](https://github.com/deepseek-ai/DeepEP/blob/main/deep_ep/buffer.py), abstracting NVLink and RDMA transports.
- **Automatic routing** selects intranode NVLink kernels versus internode RDMA paths based on `get_num_rdma_ranks()` checks in the dispatch and combine methods.
- **Topology validation** occurs at initialization via `check_nvlink_connections` in [`deep_ep/utils.py`](https://github.com/deepseek-ai/DeepEP/blob/main/deep_ep/utils.py), ensuring hardware compatibility before buffer allocation.
- **Dual-mode operation** supports both high-throughput training and low-latency inference via separate API paths, with the latter requiring explicit buffer cleaning and supporting FP8 compression.
- **Handle-based matching** ensures correct pairing of dispatch and combine operations across the distributed process group.

## Frequently Asked Questions

### How does DeepEP decide between NVLink and RDMA communication?

DeepEP checks the number of RDMA ranks via `self.runtime.get_num_rdma_ranks()` during the `dispatch` and `combine` calls. If the count exceeds 1, indicating ranks span multiple nodes, it routes to internode kernels using RDMA (IBGDA). For single-node deployments, it uses intranode NVLink kernels. This logic is implemented in [`deep_ep/buffer.py`](https://github.com/deepseek-ai/DeepEP/blob/main/deep_ep/buffer.py) lines 34-40 and 73-80.

### What is the purpose of the handle returned by dispatch operations?

The handle serves as a communication token that captures the routing metadata and buffer layout from the dispatch phase. It must be passed to the corresponding `combine` call so the kernel knows how to map reduced tensors back to their origin ranks. This design eliminates the need for recompute or retransmission of routing tables during the combine phase.

### Why does low-latency mode require buffer cleaning before each call?

Low-latency mode uses persistent RDMA buffers for minimal CPU overhead. The **`clean_low_latency_buffer`** method ensures these buffers are zero-initialized before each dispatch to prevent stale data contamination. This requirement stems from the IBGDA transport implementation where buffer reuse is critical for achieving microsecond-scale latencies in serving workloads.

### What hardware validation does DeepEP perform at startup?

During `Buffer` construction, DeepEP calls `check_nvlink_connections` (from [`deep_ep/utils.py`](https://github.com/deepseek-ai/DeepEP/blob/main/deep_ep/utils.py)) to verify that all GPU pairs in the process group can communicate over NVLink. This check queries the NVIDIA Management Library (NVML) and raises an error if the required P2P connectivity is unavailable, preventing runtime failures in the high-throughput kernels that assume direct NVLink access.