# How to Configure NVLink and RDMA in DeepEP for Hybrid Clusters

> Configure NVLink and RDMA in DeepEP for hybrid clusters. Learn to validate GPU topology, set NVSHMEM variables, and initialize Buffers for low latency and asymmetric forwarding. Optimize deep learning performance now.

- Repository: [DeepSeek/DeepEP](https://github.com/deepseek-ai/DeepEP)
- Tags: how-to-guide
- Published: 2026-04-25

---

**Configure NVLink and RDMA in DeepEP by validating GPU topology with `check_nvlink_connections()`, setting NVSHMEM environment variables such as `NVSHMEM_IB_ENABLE_IBGDA=1`, and initializing a `Buffer` with `allow_nvlink_for_low_latency_mode=True` and `allow_mnnvl=False` to enable asymmetric-domain forwarding across hybrid clusters.**

DeepEP is DeepSeek-AI's high-performance communication library for Mixture-of-Experts (MoE) models, designed to exploit both **NVLink** for high-throughput intranode GPU communication and **RDMA** for low-latency internode transfers. Configuring these transports in hybrid clusters requires coordinating runtime validation, Python constructor flags, and NVSHMEM environment variables. This guide explains how to configure NVLink and RDMA in DeepEP based on the actual implementation in the `deepseek-ai/DeepEP` repository.

## Runtime Check for NVLink Connectivity

Before communication begins, DeepEP validates the physical GPU topology to ensure optimal performance. When a `Buffer` is instantiated, the library calls `deep_ep.utils.check_nvlink_connections()` (implemented in [[`deep_ep/utils.py`](https://github.com/deepseek-ai/DeepEP/blob/main/deep_ep/utils.py)](https://github.com/deepseek-ai/DeepEP/blob/main/deep_ep/utils.py)) to verify connectivity:

- It detects GPU topology via **NVML** (NVIDIA Management Library).
- On PCIe systems, it asserts that at most two GPUs are present (pair-wise NVLink only).
- It raises a runtime error if any GPU pair inside the same node lacks NVLink connectivity.

If you need to bypass this check—for example, when testing on nodes without NVLink hardware—set the environment variable `DISABLE_NVLINK_CHECK=1` or use the `--disable-nvlink` flag demonstrated in [[`tests/test_low_latency.py`](https://github.com/deepseek-ai/DeepEP/blob/main/tests/test_low_latency.py)](https://github.com/deepseek-ai/DeepEP/blob/main/tests/test_low_latency.py#L323-L327).

## Buffer Constructor Flags for Hybrid Clusters

The `Buffer` class in [[`deep_ep/buffer.py`](https://github.com/deepseek-ai/DeepEP/blob/main/deep_ep/buffer.py)](https://github.com/deepseek-ai/DeepEP/blob/main/deep_ep/buffer.py) accepts several constructor arguments (lines 31-44) that control the interaction between NVLink and RDMA:

- **`low_latency_mode`**: When set to `True`, the buffer uses RDMA-only kernels optimized for inference latency. Set to `False` for throughput-oriented training workloads.
- **`allow_nvlink_for_low_latency_mode`**: Controls whether NVLink traffic is permitted even in low-latency mode. Set to **`True`** (default) to enable the NVLink-to-RDMA forwarding path.
- **`allow_mnnvl`**: Enables multi-node NVLink (MNNVL) detection. Set to **`False`** for standard hybrid clusters; only enable if every node has a full NVLink mesh.
- **`num_qps_per_rank`**: Specifies the number of RDMA Queue Pairs per rank, which should match your number of local experts (e.g., `24` for training, `8` for inference).

## NVSHMEM Environment Variables

DeepEP relies on NVSHMEM for underlying RDMA and NVLink management. The `Buffer.__init__` method (lines 101-127 in [[`deep_ep/buffer.py`](https://github.com/deepseek-ai/DeepEP/blob/main/deep_ep/buffer.py)](https://github.com/deepseek-ai/DeepEP/blob/main/deep_ep/buffer.py#L101-L127)) automatically exports the following environment variables when RDMA is enabled:

| Variable | Effect | Typical Value |
|----------|--------|---------------|
| `NVSHMEM_DISABLE_P2P` | Disables NVLink P2P when set to `1`. | `0` (allow NVLink when `allow_nvlink_for_low_latency_mode=True`) |
| `NVSHMEM_IB_ENABLE_IBGDA` | Enables GPU Direct RDMA (IB GDA). | `1` (required for RDMA) |
| `NVSHMEM_IBGDA_NUM_RC_PER_PE` | Number of RC QPs per peer endpoint. | Match `num_qps_per_rank` (e.g., `24`) |
| `NVSHMEM_QP_DEPTH` | Queue Pair depth for in-flight work requests. | `1024` (default) |
| `NVSHMEM_MAX_TEAMS` | Upper bound on NVSHMEM teams for SHArP. | `7` (6 default + 1 extra) |
| `NVSHMEM_DISABLE_NVLS` | Disables NVLink SHArP for exclusive RDMA usage. | `1` |
| `NVSHMEM_CUMEM_GRANULARITY` | Minimum CUDA memory allocation granularity. | `536870912` (`2**29` or 512 MiB) |
| `NVSHMEM_DISABLE_MNNVL` | Disables multi-node NVLink detection. | `1` when `allow_mnnvl=False` |

Set these variables in your job script before launching the Python process, as the `Buffer` constructor reads and applies them during initialization.

## Practical Configuration Steps for Hybrid Clusters

Follow these steps to configure DeepEP for a hybrid cluster with both NVLink and RDMA interconnects:

1. **Install NVSHMEM**: Follow the build instructions in [[`third-party/README.md`](https://github.com/deepseek-ai/DeepEP/blob/main/third-party/README.md)](https://github.com/deepseek-ai/DeepEP/blob/main/third-party/README.md) to install the required NVSHMEM library.

2. **Verify Topology**: Run `nvidia-smi topo -m` on each node to confirm NVLink connectivity between GPUs, or rely on the automatic `check_nvlink_connections()` validation.

3. **Configure Environment**: Export NVSHMEM variables before launching:

```bash
export NVSHMEM_DISABLE_P2P=0
export NVSHMEM_IB_ENABLE_IBGDA=1
export NVSHMEM_IBGDA_NUM_RC_PER_PE=24
export NVSHMEM_MAX_TEAMS=7
export NVSHMEM_DISABLE_NVLS=1
export NVSHMEM_CUMEM_GRANULARITY=$((2**29))
export NVSHMEM_DISABLE_MNNVL=1

```

4. **Initialize the Buffer**: Create the communication buffer with flags appropriate for your workload.

For **training** (throughput-oriented):

```python
from deep_ep import Buffer
import torch.distributed as dist

group = dist.new_group(list(range(dist.get_world_size())))

buf = Buffer(
    group=group,
    num_nvl_bytes=64 << 20,    # 64 MiB for NVLink

    num_rdma_bytes=256 << 20,  # 256 MiB for RDMA

    low_latency_mode=False,    # High-throughput kernels

    allow_nvlink_for_low_latency_mode=True,
    allow_mnnvl=False,
    num_qps_per_rank=24        # Match local expert count

)

```

For **inference** (latency-critical with forwarding):

```python
buf = Buffer(
    group=group,
    num_nvl_bytes=0,           # Pure RDMA mode

    num_rdma_bytes=128 << 20,
    low_latency_mode=True,     # Low-latency kernels

    allow_nvlink_for_low_latency_mode=True,  # Enable NVLink→RDMA forwarding

    allow_mnnvl=False,
    num_qps_per_rank=8         # Match inference expert count

)

```

5. **Disable Checks (Optional)**: For debugging on non-NVLink systems, launch with `--disable-nvlink` or set `DISABLE_NVLINK_CHECK=1`.

The library will automatically forward data from the NVLink domain to the RDMA domain when both are present, utilizing the asymmetric-domain kernels described in the [README.md](https://github.com/deepseek-ai/DeepEP/blob/main/README.md).

## Summary

- **Validate topology** with `check_nvlink_connections()` in [`deep_ep/utils.py`](https://github.com/deepseek-ai/DeepEP/blob/main/deep_ep/utils.py) before production runs to ensure all intranode GPU pairs are NVLink-connected.
- **Set NVSHMEM variables** such as `NVSHMEM_IB_ENABLE_IBGDA=1` and `NVSHMEM_DISABLE_MNNVL=1` to enable RDMA and disable multi-node NVLink detection in hybrid clusters.
- **Configure the Buffer** with `allow_nvlink_for_low_latency_mode=True` and `allow_mnnvl=False` to enable efficient NVLink-to-RDMA forwarding while respecting hybrid cluster constraints.
- **Match `num_qps_per_rank`** to your local expert count (typically 8 for inference, 24 for training) to optimize Queue Pair utilization.
- **Install dependencies** by building NVSHMEM according to [`third-party/README.md`](https://github.com/deepseek-ai/DeepEP/blob/main/third-party/README.md) before attempting RDMA configuration.

## Frequently Asked Questions

### What happens if my nodes lack NVLink connectivity?

DeepEP will raise a runtime error during `Buffer` initialization when `check_nvlink_connections()` detects disconnected GPU pairs. For testing on PCIe-only systems, you can bypass this by setting `DISABLE_NVLINK_CHECK=1` or using the `--disable-nvlink` flag, though this will fall back to PCIe communication with reduced performance.

### Should I use `low_latency_mode=True` for training workloads?

No. The `low_latency_mode` flag is optimized for inference scenarios requiring minimal latency. For training workloads that prioritize throughput, set `low_latency_mode=False` to use the high-throughput normal kernels that better saturate NVLink and RDMA bandwidth.

### What is the purpose of `allow_nvlink_for_low_latency_mode`?

This flag permits the low-latency RDMA kernels to utilize NVLink for intranode communication when set to `True`, creating an asymmetric forwarding path where NVLink handles intranode traffic and RDMA handles internode traffic. Even in low-latency mode, this hybrid approach often outperforms pure RDMA by leveraging the higher bandwidth of NVLink for local peer access.

### Do I need to install NVSHMEM separately?

Yes. DeepEP requires NVSHMEM as a third-party dependency for RDMA and NVLink management. You must build and install NVSHMEM separately following the instructions in [`third-party/README.md`](https://github.com/deepseek-ai/DeepEP/blob/main/third-party/README.md) before the DeepEP `Buffer` can initialize properly with RDMA support.