How to Configure NVLink and RDMA in DeepEP for Hybrid Clusters

Configure NVLink and RDMA in DeepEP by validating GPU topology with check_nvlink_connections(), setting NVSHMEM environment variables such as NVSHMEM_IB_ENABLE_IBGDA=1, and initializing a Buffer with allow_nvlink_for_low_latency_mode=True and allow_mnnvl=False to enable asymmetric-domain forwarding across hybrid clusters.

DeepEP is DeepSeek-AI's high-performance communication library for Mixture-of-Experts (MoE) models, designed to exploit both NVLink for high-throughput intranode GPU communication and RDMA for low-latency internode transfers. Configuring these transports in hybrid clusters requires coordinating runtime validation, Python constructor flags, and NVSHMEM environment variables. This guide explains how to configure NVLink and RDMA in DeepEP based on the actual implementation in the deepseek-ai/DeepEP repository.

Runtime Check for NVLink Connectivity

Before communication begins, DeepEP validates the physical GPU topology to ensure optimal performance. When a Buffer is instantiated, the library calls deep_ep.utils.check_nvlink_connections() (implemented in [deep_ep/utils.py](https://github.com/deepseek-ai/DeepEP/blob/main/deep_ep/utils.py)) to verify connectivity:

• It detects GPU topology via NVML (NVIDIA Management Library).
• On PCIe systems, it asserts that at most two GPUs are present (pair-wise NVLink only).
• It raises a runtime error if any GPU pair inside the same node lacks NVLink connectivity.

If you need to bypass this check—for example, when testing on nodes without NVLink hardware—set the environment variable DISABLE_NVLINK_CHECK=1 or use the --disable-nvlink flag demonstrated in [tests/test_low_latency.py](https://github.com/deepseek-ai/DeepEP/blob/main/tests/test_low_latency.py#L323-L327).

Buffer Constructor Flags for Hybrid Clusters

The Buffer class in [deep_ep/buffer.py](https://github.com/deepseek-ai/DeepEP/blob/main/deep_ep/buffer.py) accepts several constructor arguments (lines 31-44) that control the interaction between NVLink and RDMA:

• low_latency_mode: When set to True, the buffer uses RDMA-only kernels optimized for inference latency. Set to False for throughput-oriented training workloads.
• allow_nvlink_for_low_latency_mode: Controls whether NVLink traffic is permitted even in low-latency mode. Set to True (default) to enable the NVLink-to-RDMA forwarding path.
• allow_mnnvl: Enables multi-node NVLink (MNNVL) detection. Set to False for standard hybrid clusters; only enable if every node has a full NVLink mesh.
• num_qps_per_rank: Specifies the number of RDMA Queue Pairs per rank, which should match your number of local experts (e.g., 24 for training, 8 for inference).

NVSHMEM Environment Variables

DeepEP relies on NVSHMEM for underlying RDMA and NVLink management. The Buffer.__init__ method (lines 101-127 in [deep_ep/buffer.py](https://github.com/deepseek-ai/DeepEP/blob/main/deep_ep/buffer.py#L101-L127)) automatically exports the following environment variables when RDMA is enabled:

Variable	Effect	Typical Value
NVSHMEM_DISABLE_P2P	Disables NVLink P2P when set to 1.	0 (allow NVLink when allow_nvlink_for_low_latency_mode=True)
NVSHMEM_IB_ENABLE_IBGDA	Enables GPU Direct RDMA (IB GDA).	1 (required for RDMA)
NVSHMEM_IBGDA_NUM_RC_PER_PE	Number of RC QPs per peer endpoint.	Match num_qps_per_rank (e.g., 24)
NVSHMEM_QP_DEPTH	Queue Pair depth for in-flight work requests.	1024 (default)
NVSHMEM_MAX_TEAMS	Upper bound on NVSHMEM teams for SHArP.	7 (6 default + 1 extra)
NVSHMEM_DISABLE_NVLS	Disables NVLink SHArP for exclusive RDMA usage.	1
NVSHMEM_CUMEM_GRANULARITY	Minimum CUDA memory allocation granularity.	536870912 (2**29 or 512 MiB)
NVSHMEM_DISABLE_MNNVL	Disables multi-node NVLink detection.	1 when allow_mnnvl=False

Set these variables in your job script before launching the Python process, as the Buffer constructor reads and applies them during initialization.

Practical Configuration Steps for Hybrid Clusters

Follow these steps to configure DeepEP for a hybrid cluster with both NVLink and RDMA interconnects:

1. Install NVSHMEM: Follow the build instructions in [third-party/README.md](https://github.com/deepseek-ai/DeepEP/blob/main/third-party/README.md) to install the required NVSHMEM library.

2. Verify Topology: Run nvidia-smi topo -m on each node to confirm NVLink connectivity between GPUs, or rely on the automatic check_nvlink_connections() validation.

3. Configure Environment: Export NVSHMEM variables before launching:

export NVSHMEM_DISABLE_P2P=0
export NVSHMEM_IB_ENABLE_IBGDA=1
export NVSHMEM_IBGDA_NUM_RC_PER_PE=24
export NVSHMEM_MAX_TEAMS=7
export NVSHMEM_DISABLE_NVLS=1
export NVSHMEM_CUMEM_GRANULARITY=$((2**29))
export NVSHMEM_DISABLE_MNNVL=1

4. Initialize the Buffer: Create the communication buffer with flags appropriate for your workload.

For training (throughput-oriented):

from deep_ep import Buffer
import torch.distributed as dist

group = dist.new_group(list(range(dist.get_world_size())))

buf = Buffer(
    group=group,
    num_nvl_bytes=64 << 20,    # 64 MiB for NVLink

    num_rdma_bytes=256 << 20,  # 256 MiB for RDMA

    low_latency_mode=False,    # High-throughput kernels

    allow_nvlink_for_low_latency_mode=True,
    allow_mnnvl=False,
    num_qps_per_rank=24        # Match local expert count

)

For inference (latency-critical with forwarding):

buf = Buffer(
    group=group,
    num_nvl_bytes=0,           # Pure RDMA mode

    num_rdma_bytes=128 << 20,
    low_latency_mode=True,     # Low-latency kernels

    allow_nvlink_for_low_latency_mode=True,  # Enable NVLink→RDMA forwarding

    allow_mnnvl=False,
    num_qps_per_rank=8         # Match inference expert count

)

5. Disable Checks (Optional): For debugging on non-NVLink systems, launch with --disable-nvlink or set DISABLE_NVLINK_CHECK=1.

The library will automatically forward data from the NVLink domain to the RDMA domain when both are present, utilizing the asymmetric-domain kernels described in the README.md.

Summary

• Validate topology with check_nvlink_connections() in deep_ep/utils.py before production runs to ensure all intranode GPU pairs are NVLink-connected.
• Set NVSHMEM variables such as NVSHMEM_IB_ENABLE_IBGDA=1 and NVSHMEM_DISABLE_MNNVL=1 to enable RDMA and disable multi-node NVLink detection in hybrid clusters.
• Configure the Buffer with allow_nvlink_for_low_latency_mode=True and allow_mnnvl=False to enable efficient NVLink-to-RDMA forwarding while respecting hybrid cluster constraints.
• Match num_qps_per_rank to your local expert count (typically 8 for inference, 24 for training) to optimize Queue Pair utilization.
• Install dependencies by building NVSHMEM according to third-party/README.md before attempting RDMA configuration.

Frequently Asked Questions

What happens if my nodes lack NVLink connectivity?

DeepEP will raise a runtime error during Buffer initialization when check_nvlink_connections() detects disconnected GPU pairs. For testing on PCIe-only systems, you can bypass this by setting DISABLE_NVLINK_CHECK=1 or using the --disable-nvlink flag, though this will fall back to PCIe communication with reduced performance.

Should I use low_latency_mode=True for training workloads?

No. The low_latency_mode flag is optimized for inference scenarios requiring minimal latency. For training workloads that prioritize throughput, set low_latency_mode=False to use the high-throughput normal kernels that better saturate NVLink and RDMA bandwidth.

What is the purpose of allow_nvlink_for_low_latency_mode?

This flag permits the low-latency RDMA kernels to utilize NVLink for intranode communication when set to True, creating an asymmetric forwarding path where NVLink handles intranode traffic and RDMA handles internode traffic. Even in low-latency mode, this hybrid approach often outperforms pure RDMA by leveraging the higher bandwidth of NVLink for local peer access.

Do I need to install NVSHMEM separately?

Yes. DeepEP requires NVSHMEM as a third-party dependency for RDMA and NVLink management. You must build and install NVSHMEM separately following the instructions in third-party/README.md before the DeepEP Buffer can initialize properly with RDMA support.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}