# DeepEP Buffer Class API for Expert Parallel Communication: Architecture and Usage

> Explore the DeepEP Buffer class API for efficient expert parallel communication. This Python interface unifies CUDA NVLink and RDMA resources for low latency.

- Repository: [DeepSeek/DeepEP](https://github.com/deepseek-ai/DeepEP)
- Tags: api-reference
- Published: 2026-04-25

---

**The `deep_ep.buffer.Buffer` class provides a unified Python interface that encapsulates all CUDA, NVLink, and RDMA resources required for high-throughput and low-latency expert-parallel communication in Mixture-of-Experts (MoE) models.**

The `Buffer` class is the central abstraction in the [deepseek-ai/DeepEP](https://github.com/deepseek-ai/DeepEP) repository, designed to orchestrate token dispatch and combine operations across distributed GPU nodes with minimal overhead. Understanding the **DeepEP Buffer class API for expert parallel communication** is essential for implementing scalable MoE training and inference pipelines on clusters utilizing NVLink and InfiniBand hardware.

## Architecture and Initialization

The `Buffer` class defined in [`deep_ep/buffer.py`](https://github.com/deepseek-ai/DeepEP/blob/main/deep_ep/buffer.py) serves as a thin Python façade over a high-performance C++ runtime (`deep_ep_cpp.Buffer`), managing the lifecycle of communication buffers and kernel execution streams.

### Construction and Resource Allocation

When instantiating a `Buffer` object, the constructor (`__init__` at lines 32-78) performs several critical setup operations:

- **Validates NVLink topology** across the participating process group using `check_nvlink_connections`
- **Allocates device memory** for both NVLink (intranode) and RDMA (internode) communication buffers according to the specified `num_nvl_bytes` and `num_rdma_bytes` parameters
- **Exchanges IPC handles** and NVSHMEM unique IDs across the group, supporting both `torch.distributed` and `mpi4py` communicators
- **Sets environment variables** required for NVSHMEM operation, including `NVSHMEM_IB_ENABLE_IBGDA` for GPU-direct access
- **Synchronizes the runtime** to ensure all ranks share a consistent view of the allocated buffers

The C++ runtime object is created at lines 92-95 and stored as `self.runtime`, providing access to the underlying kernel implementations.

### Configuration and Layout Helpers

The class provides static configuration helpers to optimize kernel selection based on group size:

- **`get_dispatch_config`** and **`get_combine_config`** (lines 44-61) return performance-tuned `Config` objects specifying kernel parameters for different scales of parallelism
- **`get_dispatch_layout`** (lines 92-100) pre-computes the token-to-rank mapping, returning `num_tokens_per_rank` and `is_token_in_rank` tensors that describe how input tokens should be partitioned across experts before the actual data movement occurs

## Core Communication Workflow

The standard expert-parallel communication pattern follows a dispatch-compute-combine cycle, implemented through high-throughput all-to-all operations.

### Dispatch Operation

The **`dispatch`** method (lines 122-172) performs the forward communication that splits tokens to their target experts:

- Optionally accepts a pre-computed layout from `get_dispatch_layout`, or computes it internally from `topk_idx` indices
- Executes intranode all-to-all over NVLink or internode transfers over RDMA based on the target expert locations
- Returns a **handle** object containing prefix-sum matrices and source indices, along with the received tokens, indices, weights, and token counts per rank
- Provides a CUDA `event` for stream synchronization and an `EventOverlap` hook for custom synchronization logic

### Combine Operation

The **`combine`** method (also lines 122-172) handles the backward reduction of expert outputs:

- Accepts the handle returned by the corresponding `dispatch` call to properly map reduced values back to source tokens
- Performs addition-based reduction of the received tensors
- Optionally applies per-token top-k weights and bias terms during the reduction
- Supports both high-throughput NVLink and RDMA communication paths automatically selected by the runtime

## Low-Latency Communication Path

For latency-critical inference scenarios, DeepEP provides IB-GDA (InfiniBand GPU Direct Access) kernels that bypass intermediate buffer copies.

### IB-GDA Kernels

The low-latency API (lines 332-382) enables direct RDMA transfers with minimal CPU involvement:

- **`low_latency_dispatch`** streams tokens directly into pre-registered RDMA buffers without intermediate staging, returning packed tensors and communication handles
- **`low_latency_combine`** reduces expert results back to the original token positions, supporting **FP8 quantization** when `use_fp8=True` is specified
- Validates NVSHMEM queue pair depth against `self.nvshmem_qp_depth` to ensure sufficient hardware resources for the transfer size

### Dynamic Rank Masking

The class provides mask-buffer utilities (lines 624-684) for fault tolerance and selective participation:

- **`low_latency_update_mask_buffer`** allows disabling specific ranks (e.g., failed nodes) by setting mask bits without tearing down the entire communication context
- **`low_latency_query_mask_buffer`** exports the current mask status to a CPU tensor showing which ranks are active (0) or masked (1)
- **`low_latency_clean_mask_buffer`** resets mask states for recovery scenarios

## Advanced Resource Management

Beyond the core communication methods, the `Buffer` class exposes utilities for fine-grained control:

- **`get_comm_stream`** returns the dedicated CUDA stream used for all communication operations, allowing explicit synchronization with computation streams
- **`get_local_buffer_tensor`** exposes the raw NVLink or RDMA memory regions as PyTorch tensors for custom data manipulation or inspection
- **`capture`** provides an `EventOverlap` wrapper (defined in [`deep_ep/utils.py`](https://github.com/deepseek-ai/DeepEP/blob/main/deep_ep/utils.py)) for capturing CUDA events to synchronize communication with other GPU work

### Explicit Resource Cleanup

While the class implements a destructor for automatic cleanup, the **`destroy`** method (lines 38-46) allows explicit release of NVSHMEM resources and buffer memory, critical for long-running applications or dynamic reconfiguration of process groups.

## Code Examples

### Basic Dispatch and Combine

This example demonstrates standard high-throughput communication for MoE training:

```python
import torch
import torch.distributed as dist
from deep_ep.buffer import Buffer

# Initialize within an existing torch.distributed process group

group = dist.group.WORLD

# Create buffer: 2 MiB NVLink, 8 MiB RDMA

buf = Buffer(group, num_nvl_bytes=2 << 20, num_rdma_bytes=8 << 20)

# Prepare input: 1024 tokens, 4096 dimensions, 16 experts, top-2 routing

tokens = torch.randn(1024, 4096, dtype=torch.bfloat16, device='cuda')
topk_idx = torch.randint(0, 16, (1024, 2), dtype=torch.int64)

# Dispatch to experts

recv_tokens, recv_idx, recv_weights, recv_counts, handle, event = \
    buf.dispatch(tokens, topk_idx=topk_idx)

# ... expert computation happens here ...

# Combine results back using the handle from dispatch

combined, combined_weights, _ = buf.combine(recv_tokens, handle, recv_weights)

# Cleanup

buf.destroy()

```

### Low-Latency Inference with FP8

For serving scenarios requiring minimal latency:

```python
import torch
import torch.distributed as dist
from deep_ep.buffer import Buffer

group = dist.group.WORLD

# Enable low-latency mode with 16 MiB RDMA buffer

buf = Buffer(group, num_rdma_bytes=16 << 20, low_latency_mode=True)

x = torch.randn(512, 4096, dtype=torch.bfloat16, device='cuda')
topk_idx = torch.randint(0, 32, (512, 2), dtype=torch.int64)

# Dispatch with FP8 quantization

packed_x, counts, handle, event, hook = \
    buf.low_latency_dispatch(
        x, topk_idx, 
        num_max_dispatch_tokens_per_rank=64,
        num_experts=32, 
        use_fp8=True
    )

# Expert processing on packed_x...

# Combine with weights

weights = torch.randn_like(packed_x[0])
combined, event, hook = buf.low_latency_combine(
    packed_x[0], topk_idx, weights, handle
)

```

### Dynamic Rank Masking

Handling node failures during distributed inference:

```python

# Mask rank 3 (e.g., node failure)

buf.low_latency_update_mask_buffer(rank_to_mask=3, mask=True)

# Check current status

mask_status = torch.empty(buf.group_size, dtype=torch.int, device='cpu')
buf.low_latency_query_mask_buffer(mask_status)

# Unmask when recovered

buf.low_latency_update_mask_buffer(rank_to_mask=3, mask=False)

# Clean up mask state

buf.low_latency_clean_mask_buffer()

```

## Summary

The `deep_ep.buffer.Buffer` class serves as the single-point API for expert-parallel communication in DeepEP, providing:

- **Unified resource management** for NVLink and RDMA buffers with automatic topology detection and IPC handle exchange
- **High-throughput dispatch and combine** operations via `dispatch` and `combine` methods using optimized all-to-all kernels
- **Low-latency inference paths** through `low_latency_dispatch` and `low_latency_combine` with direct IB-GDA support and optional FP8 quantization
- **Fault tolerance mechanisms** via dynamic rank masking utilities that allow selective participation without context teardown
- **Explicit lifecycle control** through the `destroy` method and accessor functions for communication streams and raw buffer tensors

## Frequently Asked Questions

### What is the difference between the standard dispatch/combine methods and the low-latency variants?

The standard **`dispatch`** and **`combine`** methods (defined at lines 122-172 of [`deep_ep/buffer.py`](https://github.com/deepseek-ai/DeepEP/blob/main/deep_ep/buffer.py)) implement high-throughput all-to-all communication optimized for training workloads, utilizing internal buffer staging for NVLink and RDMA transfers. In contrast, **`low_latency_dispatch`** and **`low_latency_combine`** (lines 332-382) use IB-GDA (GPU Direct Access) to stream data directly into pre-registered RDMA buffers without intermediate copies, reducing latency for inference services. The low-latency path also supports FP8 quantization and requires specific NVSHMEM queue pair depth validation.

### How does the Buffer class handle communication resource cleanup?

The `Buffer` class provides the **`destroy`** method (lines 38-46) for explicit resource deallocation, which tears down the underlying C++ runtime and releases NVSHMEM resources and allocated GPU memory. While the class implements a Python destructor for automatic cleanup, explicit calls to `destroy()` are recommended in long-running applications or when dynamically reconfiguring process groups to ensure immediate release of scarce RDMA and NVLink resources.

### What is the purpose of the handle returned by dispatch operations?

The **handle** returned by both `dispatch` and `low_latency_dispatch` is an opaque object containing metadata required for the subsequent combine operation, specifically prefix-sum matrices and source token indices that map expert outputs back to their original positions in the input tensor. According to the implementation in [`deep_ep/buffer.py`](https://github.com/deepseek-ai/DeepEP/blob/main/deep_ep/buffer.py), this handle must be preserved and passed to the corresponding `combine` or `low_latency_combine` call to ensure correct reduction and reordering of results across the distributed expert parallelism group.

### How can I disable specific ranks during communication without restarting the job?

The `Buffer` class provides mask-buffer utilities (lines 624-684) for dynamic rank exclusion. Use **`low_latency_update_mask_buffer(rank_to_mask, mask=True)`** to disable a specific rank (e.g., a failed node), and query the current status with **`low_latency_query_mask_buffer(output_tensor)`**. This allows the communication group to continue operating with reduced capacity rather than failing entirely. The **`low_latency_clean_mask_buffer`** method resets all masks when the excluded ranks recover and rejoin the computation.