How to Configure IBGDA in DeepEP: Complete Setup Guide for InfiniBand GPU Direct Access

To configure IBGDA in DeepEP, enable the NVIDIA driver module options NVreg_EnableStreamMemOPs=1 and NVreg_RegistryDwords="PeerMappingOverride=1;", install NVSHMEM ≥3.3.9, and instantiate deep_ep.Buffer with low_latency_mode=True to automatically inject the required environment variables.

Configuring InfiniBand GPU Direct Async (IBGDA) in the deepseek-ai/DeepEP repository enables zero-copy RDMA communication directly from GPU threads, eliminating CPU bottlenecks in inter-node collective operations. This configuration requires coordination across hardware drivers, system libraries, and the DeepEP Python runtime. The setup involves three distinct layers: kernel driver parameters, NVSHMEM library installation, and runtime environment variables set during Buffer initialization.

Prerequisites for IBGDA Configuration

Before enabling IBGDA in DeepEP, your system must satisfy specific hardware and software requirements.

Hardware and Driver Requirements

IBGDA requires GPUs connected via NVLink for intra-node communication and attached to RDMA-capable InfiniBand NICs for inter-node traffic. According to the third-party/README.md (lines 36-50), you must configure the NVIDIA kernel module with specific registry options to expose the necessary memory mapping capabilities.

NVSHMEM Installation

Install NVSHMEM version 3.3.9 or later, which provides the underlying communication primitives that DeepEP uses. The third-party/README.md documentation provides detailed build instructions for different IBGDA modes.

Driver-Level IBGDA Activation

The traditional IBGDA configuration requires modifying the NVIDIA kernel module parameters. Create a configuration file to enable stream memory operations and peer mapping overrides:

# /etc/modprobe.d/nvidia.conf

options nvidia NVreg_EnableStreamMemOPs=1 NVreg_RegistryDwords="PeerMappingOverride=1;"

After saving the configuration, rebuild the initial RAM filesystem and reboot the system to apply the changes:

sudo update-initramfs -u
sudo reboot

This configuration enables the driver-level support necessary for GPUDirect Async operations, allowing GPU threads to directly initiate InfiniBand verbs without CPU intervention.

Alternative: CPU-Assisted IBGDA with GDRCopy

If your system policies prohibit modifying NVIDIA kernel module options, you can use the CPU-assisted IBGDA path via GDRCopy. This method incurs a small performance penalty but functions on locked-down systems.

Install GDRCopy through your package manager or from source, then load the kernel module:

# Install GDRCopy (adjust for your distribution)

sudo apt-get install gdrcopy
sudo modprobe gdrdrv

This fallback path is documented in the third-party/README.md as an alternative when the NVreg_EnableStreamMemOPs driver option is unavailable.

DeepEP Runtime Configuration

DeepEP automatically configures NVSHMEM environment variables when you create a Buffer instance with low-latency mode enabled. In deep_ep/buffer.py (lines 106-112), the constructor injects the following variables before initializing the underlying NVSHMEM runtime:

# deep_ep/buffer.py – lines 106-112

os.environ['NVSHMEM_DISABLE_P2P'] = '0' if allow_nvlink_for_low_latency_mode else '1'
os.environ['NVSHMEM_IB_ENABLE_IBGDA'] = '1'                     # Enable IBGDA

os.environ['NVSHMEM_IBGDA_NUM_RC_PER_PE'] = f'{num_qps_per_rank}'  # RC QPs per rank

os.environ['NVSHMEM_QP_DEPTH'] = str(self.nvshmem_qp_depth)   # Queue pair depth

os.environ['NVSHMEM_MAX_TEAMS'] = '7'                         # Memory optimization

os.environ['NVSHMEM_DISABLE_NVLS'] = '1'                       # Disable NVLink SHArP

os.environ['NVSHMEM_CUMEM_GRANULARITY'] = f'{2 ** 29}'        # ≥ 256 MiB requirement

These environment variables configure NVSHMEM to use IBGDA transport, allocate sufficient queue pairs for the connection count, and ensure proper memory granularity for GPUDirect operations.

Low-Level IBGDA Implementation

The actual IBGDA device primitives reside in csrc/kernels/ibgda_device.cuh. These CUDA device functions—including ibgda_post_send and ibgda_submit_requests—map directly to InfiniBand verbs and are invoked from the inter-node communication kernels in csrc/kernels/internode.cu and csrc/kernels/internode_ll.cu. The compile-time constants for queue pair depth and other parameters are defined in csrc/kernels/configs.cuh.

Complete Configuration Example

The following example demonstrates creating a low-latency buffer with IBGDA enabled:

import deep_ep
import torch

# Number of QPs per rank – tune based on NIC capability (e.g., 4)

num_qps_per_rank = 4

# Enable low-latency (IBGDA) mode

buf = deep_ep.Buffer(
    rank=0,
    group_size=8,
    num_nvl_bytes=1 << 30,          # 1 GiB NVLink buffer

    num_rdma_bytes=1 << 30,         # 1 GiB RDMA buffer

    low_latency_mode=True,          # Turn on IBGDA

    num_qps_per_rank=num_qps_per_rank,
    allow_nvlink_for_low_latency_mode=True,
)

# Use the communication stream for async ops

comm_stream = buf.get_comm_stream()
torch.cuda.synchronize(comm_stream.device)

To verify that IBGDA is active after buffer creation, check the NVSHMEM runtime information:

import subprocess

# After Buffer construction, NVSHMEM reports IBGDA support

info = subprocess.check_output(['nvshmem-info', '-a']).decode()
print('IBGDA enabled:', 'IBGDA' in info)

Successful configuration produces output containing IBGDA support: enabled.

Summary

• Driver configuration requires setting NVreg_EnableStreamMemOPs=1 and PeerMappingOverride=1 in the NVIDIA kernel module options, or alternatively installing GDRCopy for CPU-assisted mode.
• Library dependency mandates NVSHMEM version 3.3.9 or later for IBGDA support.
• Runtime activation occurs automatically in deep_ep/buffer.py when constructing a Buffer with low_latency_mode=True, which sets NVSHMEM_IB_ENABLE_IBGDA=1 and related environment variables.
• Device implementation uses the primitives in csrc/kernels/ibgda_device.cuh for GPU-initiated RDMA operations.

Frequently Asked Questions

What is the minimum NVSHMEM version required for IBGDA support in DeepEP?

DeepEP requires NVSHMEM 3.3.9 or later to utilize IBGDA features. Earlier versions lack the necessary GPUDirect Async integration and environment variable interfaces used by the buffer.py runtime configuration.

Can I use IBGDA without modifying NVIDIA kernel module parameters?

Yes, if driver modification is restricted, you can use the CPU-assisted IBGDA path by installing and loading the GDRCopy kernel module (gdrdrv). This allows GPU memory access for RDMA operations without the NVreg_EnableStreamMemOPs driver option, though with slightly higher latency than the direct driver-enabled path.

How does DeepEP handle IBGDA queue pair configuration?

DeepEP configures queue pairs through the num_qps_per_rank parameter passed to the Buffer constructor. The runtime sets NVSHMEM_IBGDA_NUM_RC_PER_PE to this value and configures NVSHMEM_QP_DEPTH based on the buffer's nvshmem_qp_depth attribute, ensuring sufficient connection resources for your cluster size.

Where are the IBGDA device primitives implemented in the DeepEP source code?

The low-level IBGDA device functions are implemented in csrc/kernels/ibgda_device.cuh. These CUDA functions interface directly with InfiniBand hardware and are called from the inter-node kernels in csrc/kernels/internode.cu and csrc/kernels/internode_ll.cu to perform zero-copy RDMA transfers initiated by GPU threads.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}