# How to Configure IBGDA in DeepEP: Complete Setup Guide for InfiniBand GPU Direct Access

> Configure IBGDA in DeepEP with this guide. Enable NVIDIA driver options, install NVSHMEM, and optimize your setup for low-latency GPU Direct Access. Learn how now.

- Repository: [DeepSeek/DeepEP](https://github.com/deepseek-ai/DeepEP)
- Tags: how-to-guide
- Published: 2026-04-25

---

**To configure IBGDA in DeepEP, enable the NVIDIA driver module options `NVreg_EnableStreamMemOPs=1` and `NVreg_RegistryDwords="PeerMappingOverride=1;"`, install NVSHMEM ≥3.3.9, and instantiate `deep_ep.Buffer` with `low_latency_mode=True` to automatically inject the required environment variables.**

Configuring InfiniBand GPU Direct Async (IBGDA) in the `deepseek-ai/DeepEP` repository enables zero-copy RDMA communication directly from GPU threads, eliminating CPU bottlenecks in inter-node collective operations. This configuration requires coordination across hardware drivers, system libraries, and the DeepEP Python runtime. The setup involves three distinct layers: kernel driver parameters, NVSHMEM library installation, and runtime environment variables set during `Buffer` initialization.

## Prerequisites for IBGDA Configuration

Before enabling IBGDA in DeepEP, your system must satisfy specific hardware and software requirements.

### Hardware and Driver Requirements

IBGDA requires GPUs connected via NVLink for intra-node communication and attached to RDMA-capable InfiniBand NICs for inter-node traffic. According to the [`third-party/README.md`](https://github.com/deepseek-ai/DeepEP/blob/main/third-party/README.md) (lines 36-50), you must configure the NVIDIA kernel module with specific registry options to expose the necessary memory mapping capabilities.

### NVSHMEM Installation

Install **NVSHMEM version 3.3.9 or later**, which provides the underlying communication primitives that DeepEP uses. The [`third-party/README.md`](https://github.com/deepseek-ai/DeepEP/blob/main/third-party/README.md) documentation provides detailed build instructions for different IBGDA modes.

## Driver-Level IBGDA Activation

The traditional IBGDA configuration requires modifying the NVIDIA kernel module parameters. Create a configuration file to enable stream memory operations and peer mapping overrides:

```bash

# /etc/modprobe.d/nvidia.conf

options nvidia NVreg_EnableStreamMemOPs=1 NVreg_RegistryDwords="PeerMappingOverride=1;"

```

After saving the configuration, rebuild the initial RAM filesystem and reboot the system to apply the changes:

```bash
sudo update-initramfs -u
sudo reboot

```

This configuration enables the driver-level support necessary for GPUDirect Async operations, allowing GPU threads to directly initiate InfiniBand verbs without CPU intervention.

## Alternative: CPU-Assisted IBGDA with GDRCopy

If your system policies prohibit modifying NVIDIA kernel module options, you can use the CPU-assisted IBGDA path via **GDRCopy**. This method incurs a small performance penalty but functions on locked-down systems.

Install GDRCopy through your package manager or from source, then load the kernel module:

```bash

# Install GDRCopy (adjust for your distribution)

sudo apt-get install gdrcopy
sudo modprobe gdrdrv

```

This fallback path is documented in the [`third-party/README.md`](https://github.com/deepseek-ai/DeepEP/blob/main/third-party/README.md) as an alternative when the `NVreg_EnableStreamMemOPs` driver option is unavailable.

## DeepEP Runtime Configuration

DeepEP automatically configures NVSHMEM environment variables when you create a `Buffer` instance with low-latency mode enabled. In [`deep_ep/buffer.py`](https://github.com/deepseek-ai/DeepEP/blob/main/deep_ep/buffer.py) (lines 106-112), the constructor injects the following variables **before** initializing the underlying NVSHMEM runtime:

```python

# deep_ep/buffer.py – lines 106-112

os.environ['NVSHMEM_DISABLE_P2P'] = '0' if allow_nvlink_for_low_latency_mode else '1'
os.environ['NVSHMEM_IB_ENABLE_IBGDA'] = '1'                     # Enable IBGDA

os.environ['NVSHMEM_IBGDA_NUM_RC_PER_PE'] = f'{num_qps_per_rank}'  # RC QPs per rank

os.environ['NVSHMEM_QP_DEPTH'] = str(self.nvshmem_qp_depth)   # Queue pair depth

os.environ['NVSHMEM_MAX_TEAMS'] = '7'                         # Memory optimization

os.environ['NVSHMEM_DISABLE_NVLS'] = '1'                       # Disable NVLink SHArP

os.environ['NVSHMEM_CUMEM_GRANULARITY'] = f'{2 ** 29}'        # ≥ 256 MiB requirement

```

These environment variables configure NVSHMEM to use IBGDA transport, allocate sufficient queue pairs for the connection count, and ensure proper memory granularity for GPUDirect operations.

## Low-Level IBGDA Implementation

The actual IBGDA device primitives reside in `csrc/kernels/ibgda_device.cuh`. These CUDA device functions—including `ibgda_post_send` and `ibgda_submit_requests`—map directly to InfiniBand verbs and are invoked from the inter-node communication kernels in `csrc/kernels/internode.cu` and `csrc/kernels/internode_ll.cu`. The compile-time constants for queue pair depth and other parameters are defined in `csrc/kernels/configs.cuh`.

## Complete Configuration Example

The following example demonstrates creating a low-latency buffer with IBGDA enabled:

```python
import deep_ep
import torch

# Number of QPs per rank – tune based on NIC capability (e.g., 4)

num_qps_per_rank = 4

# Enable low-latency (IBGDA) mode

buf = deep_ep.Buffer(
    rank=0,
    group_size=8,
    num_nvl_bytes=1 << 30,          # 1 GiB NVLink buffer

    num_rdma_bytes=1 << 30,         # 1 GiB RDMA buffer

    low_latency_mode=True,          # Turn on IBGDA

    num_qps_per_rank=num_qps_per_rank,
    allow_nvlink_for_low_latency_mode=True,
)

# Use the communication stream for async ops

comm_stream = buf.get_comm_stream()
torch.cuda.synchronize(comm_stream.device)

```

To verify that IBGDA is active after buffer creation, check the NVSHMEM runtime information:

```python
import subprocess

# After Buffer construction, NVSHMEM reports IBGDA support

info = subprocess.check_output(['nvshmem-info', '-a']).decode()
print('IBGDA enabled:', 'IBGDA' in info)

```

Successful configuration produces output containing `IBGDA support: enabled`.

## Summary

- **Driver configuration** requires setting `NVreg_EnableStreamMemOPs=1` and `PeerMappingOverride=1` in the NVIDIA kernel module options, or alternatively installing GDRCopy for CPU-assisted mode.
- **Library dependency** mandates NVSHMEM version 3.3.9 or later for IBGDA support.
- **Runtime activation** occurs automatically in [`deep_ep/buffer.py`](https://github.com/deepseek-ai/DeepEP/blob/main/deep_ep/buffer.py) when constructing a `Buffer` with `low_latency_mode=True`, which sets `NVSHMEM_IB_ENABLE_IBGDA=1` and related environment variables.
- **Device implementation** uses the primitives in `csrc/kernels/ibgda_device.cuh` for GPU-initiated RDMA operations.

## Frequently Asked Questions

### What is the minimum NVSHMEM version required for IBGDA support in DeepEP?

DeepEP requires **NVSHMEM 3.3.9 or later** to utilize IBGDA features. Earlier versions lack the necessary GPUDirect Async integration and environment variable interfaces used by the [`buffer.py`](https://github.com/deepseek-ai/DeepEP/blob/main/buffer.py) runtime configuration.

### Can I use IBGDA without modifying NVIDIA kernel module parameters?

Yes, if driver modification is restricted, you can use the **CPU-assisted IBGDA** path by installing and loading the **GDRCopy** kernel module (`gdrdrv`). This allows GPU memory access for RDMA operations without the `NVreg_EnableStreamMemOPs` driver option, though with slightly higher latency than the direct driver-enabled path.

### How does DeepEP handle IBGDA queue pair configuration?

DeepEP configures queue pairs through the `num_qps_per_rank` parameter passed to the `Buffer` constructor. The runtime sets `NVSHMEM_IBGDA_NUM_RC_PER_PE` to this value and configures `NVSHMEM_QP_DEPTH` based on the buffer's `nvshmem_qp_depth` attribute, ensuring sufficient connection resources for your cluster size.

### Where are the IBGDA device primitives implemented in the DeepEP source code?

The low-level IBGDA device functions are implemented in `csrc/kernels/ibgda_device.cuh`. These CUDA functions interface directly with InfiniBand hardware and are called from the inter-node kernels in `csrc/kernels/internode.cu` and `csrc/kernels/internode_ll.cu` to perform zero-copy RDMA transfers initiated by GPU threads.