How to Configure DeepEP Low-Latency Mode QP (Queue Pair) for RDMA Optimization

To configure DeepEP low-latency mode Queue Pairs, set num_qps_per_rank equal to the number of local experts (num_experts // world_size) when creating the buffer with low_latency_mode=True.

DeepEP is an open-source communication library designed by DeepSeek for Mixture-of-Experts (MoE) training and inference. When operating in low-latency mode, the library utilizes pure RDMA through an IBGDA (Infiniband Gather-DMA) transport, where each Queue Pair (QP) provides a dedicated RDMA channel to a local expert. Configuring the DeepEP low-latency mode QP correctly ensures minimal round-trip latency during dispatch and combine operations.

Understanding the Queue Pair Requirement

DeepEP's low-latency architecture creates an IBGDA transport that maps each local expert to exactly one QP. This design eliminates contention by allowing the dispatch kernel to issue single RDMA writes per token without extra atomic operations.

The One-to-One Mapping Rule

The most critical constraint when configuring DeepEP low-latency mode QP settings is the mandatory 1-to-1 mapping between QPs and local experts. According to the repository's README.md (lines 56-58), "the QP number must be equal to the number of the local experts".

In deep_ep/buffer.py, the Buffer class initialization enforces this relationship. When low_latency_mode=True is specified, the constructor expects num_qps_per_rank to match the expert count per rank, calculated as:

num_qps_per_rank = num_experts // world_size

Why QP Count Matters for Performance

Each QP represents a dedicated RDMA channel to an expert's buffer. With the correct DeepEP low-latency mode QP configuration:

• Direct addressing: The dispatch kernel addresses experts directly via dedicated QPs
• Reduced latency: Eliminates fallback to generic RDMA paths that increase round-trip time
• Ordering guarantees: Maintains proper memory ordering without additional synchronization

If the QP count mismatches the local expert count, the library either raises an assertion error or silently falls back to higher-latency transport mechanisms, potentially violating performance expectations established in the low-latency design.

Step-by-Step QP Configuration

Configuring the Queue Pairs requires three specific actions during buffer initialization.

Calculate Local Experts

First, determine the number of experts residing on the current rank. DeepEP assumes uniform expert distribution across the world size:

import torch.distributed as dist

world_size = dist.get_world_size()
local_experts = num_experts // world_size  # Must divide evenly

Set low_latency_mode and num_qps_per_rank

In deep_ep/buffer.py (lines 36-38), the Buffer.__init__ method accepts the low_latency_mode boolean and num_qps_per_rank integer. You must set these explicitly:

import deep_ep

buffer = deep_ep.Buffer(
    group=process_group,
    num_nvl_bytes=0,  # NVLink not used in pure low-latency mode

    num_rdma_bytes=deep_ep.Buffer.get_low_latency_rdma_size_hint(
        num_tokens, hidden, world_size, num_experts
    ),
    low_latency_mode=True,          # Enable IBGDA transport

    num_qps_per_rank=local_experts,  # ✅ Critical: QP count = expert count

    allow_nvlink_for_low_latency_mode=True,  # Set False for PCIe-only hardware

    explicitly_destroy=True
)

As shown in tests/test_low_latency.py (lines 62-65), this pattern represents the standard initialization for low-latency workloads.

Verify NVSHMEM Environment Variables

DeepEP translates the num_qps_per_rank parameter into the NVSHMEM environment variable NVSHMEM_IBGDA_NUM_RC_PER_PE (lines 107-110 in deep_ep/buffer.py). Additionally, the code enforces a minimum QP depth via NVSHMEM_QP_DEPTH (default 1024) at lines 12-15 to prevent Work-Request slot exhaustion.

The library automatically sets NVSHMEM_IB_ENABLE_IBGDA=1, but you should verify your NVSHMEM build includes IBGDA support before runtime.

Implementation Examples

Manual Buffer Creation

For maximum control, instantiate the Buffer class directly as demonstrated in the test suite:

import deep_ep
import torch.distributed as dist

# Setup distributed environment

group = dist.new_group()
rank, world_size = dist.get_rank(), dist.get_world_size()

# Model configuration

num_tokens = 128
hidden = 7168
num_experts = 64          # Must be divisible by world_size

num_topk = 8

# Calculate RDMA buffer requirements and QP count

rdma_bytes = deep_ep.Buffer.get_low_latency_rdma_size_hint(
    num_tokens, hidden, world_size, num_experts
)
num_qps = num_experts // world_size  # 8 QPs per rank if world_size=8

# Create the low-latency buffer

buffer = deep_ep.Buffer(
    group,
    num_nvl_bytes=0,
    num_rdma_bytes=rdma_bytes,
    low_latency_mode=True,
    num_qps_per_rank=num_qps,        # Maps 1 QP per local expert

    allow_nvlink_for_low_latency_mode=True,
    explicitly_destroy=True
)

Using the get_buffer Helper

For inference servers and singleton patterns, use the get_buffer helper documented in the README.md:

from deep_ep import get_buffer

# This helper automatically calculates local experts and QP requirements

buffer = get_buffer(
    group=process_group,
    num_max_dispatch_tokens_per_rank=128,
    hidden=7168,
    num_experts=64
)

The helper performs the QP-to-expert validation internally and constructs the buffer with the correct num_qps_per_rank.

Runtime Dispatch and Combine Operations

Once configured, the DeepEP low-latency mode QP setup enables specific low-latency API methods:

# Dispatch tokens to experts using the configured QPs

recv_hidden, recv_counts, handle, event, hook = buffer.low_latency_dispatch(
    hidden_states, 
    topk_idx,
    num_max_dispatch_tokens_per_rank=128,
    num_experts=64,
    async_finish=False,
    return_recv_hook=True
)

# Combine expert outputs

combined, event2, hook2 = buffer.low_latency_combine(
    recv_hidden, 
    topk_idx, 
    topk_weights, 
    handle
)

These methods leverage the IBGDA transport established during buffer creation, utilizing the dedicated QP channels for each expert.

Troubleshooting and Validation

If you encounter initialization errors or performance degradation:

• Assertion failures: Verify that num_experts divides evenly by world_size and that num_qps_per_rank equals num_experts // world_size
• NVLink compatibility: Set allow_nvlink_for_low_latency_mode=False if running on PCIe-only hardware without NVLink bridges
• QP depth warnings: The library automatically sets NVSHMEM_QP_DEPTH=1024 minimum; do not override this to lower values

Summary

• QP Requirement: DeepEP low-latency mode requires exactly one QP per local expert (num_experts // world_size)
• Configuration Location: Set via num_qps_per_rank in deep_ep.Buffer.__init__ or through the get_buffer helper
• Environment Impact: The parameter translates to NVSHMEM_IBGDA_NUM_RC_PER_PE in the underlying NVSHMEM transport
• Validation: Mismatched QP counts trigger assertions or fallback to higher-latency RDMA paths

Frequently Asked Questions

What happens if num_qps_per_rank does not match the local expert count?

The DeepEP runtime will raise an assertion error during buffer initialization, or in some configurations, silently fall back to a generic RDMA path that increases latency and may violate ordering guarantees required for correct MoE computation.

Can I use NVLink instead of InfiniBand for low-latency mode?

Yes, but with restrictions. Set allow_nvlink_for_low_latency_mode=True to enable NVLink for low-latency operations. However, pure low-latency mode primarily targets InfiniBand RDMA via IBGDA; NVLink serves as an alternative transport when configured explicitly.

Where is the QP configuration validated in the source code?

The validation occurs in deep_ep/buffer.py (lines 107-110) where the library translates Python parameters to NVSHMEM environment variables, and in README.md (lines 56-58) where the user-level helper performs sanity checks before buffer construction.

Is there a minimum QP depth requirement for DeepEP?

Yes. According to lines 12-15 in deep_ep/buffer.py, the library enforces a minimum NVSHMEM_QP_DEPTH of 1024 to avoid Work-Request slot checks that would otherwise add latency to the critical path.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}