Does DeepEP Support Multi-Node NVLink (MNNVL)? Configuration and Usage Guide

Yes, DeepEP supports Multi-Node NVLink (MNNVL) when explicitly enabled via the allow_mnnvl flag in the Buffer class constructor or the --allow-mnnvl command-line option.

The deepseek-ai/DeepEP library provides high-performance communication primitives for Mixture-of-Experts (MoE) training. While the library defaults to RDMA for inter-node traffic, it includes optional DeepEP Multi-Node NVLink support to leverage direct GPU-to-GPU links across server boundaries when the underlying NVSHMEM library and hardware infrastructure permit it.

How MNNVL Support Works in DeepEP

DeepEP implements Multi-Node NVLink support through careful environment variable management during buffer initialization. The core logic determines whether the NVSHMEM runtime should attempt to detect cross-node NVLink connections.

The allow_mnnvl Flag in buffer.py

In deep_ep/buffer.py (lines 24-40), the Buffer class constructor accepts an allow_mnnvl boolean parameter that defaults to False. When set to True, the library leaves the NVSHMEM_DISABLE_MNNVL environment variable unset, allowing the underlying NVSHMEM runtime to detect and utilize NVLink connections across nodes. When the flag remains False, DeepEP explicitly sets NVSHMEM_DISABLE_MNNVL=1 before initializing NVSHMEM, effectively disabling MNNVL to prevent runtime errors on unsupported systems.

Environment Variable Control

The implementation directly manipulates process environment variables before NVSHMEM initialization. This approach ensures that MNNVL activation requires explicit opt-in at the application level, preventing automatic attempts to use Multi-Node NVLink on clusters where the hardware or software stack does not support it.

Enabling DeepEP Multi-Node NVLink

Activating MNNVL requires two conditions: explicit application-level configuration and a compatible NVSHMEM build. You can enable support either programmatically through the Python API or via command-line flags in the test suite.

Python API Configuration

Pass allow_mnnvl=True when instantiating the Buffer class to permit NVSHMEM to use Multi-Node NVLink:

import torch.distributed as dist
from deep_ep.buffer import Buffer

# Initialize your distributed process group

group = dist.group.WORLD

# Create buffer with MNNVL enabled

buffer = Buffer(
    group=group,
    num_nvl_bytes=64 * 1024 * 1024,      # IntrA-node NVLink buffer size

    num_rdma_bytes=128 * 1024 * 1024,    # Inter-node RDMA buffer size

    low_latency_mode=False,
    allow_mnnvl=True                   # Enable Multi-Node NVLink

)

Command-Line Interface

The test harnesses expose the --allow-mnnvl flag for validation. As implemented in tests/test_intranode.py at line 306 and tests/test_low_latency.py at line 322, you can run:

python tests/test_intranode.py --allow-mnnvl
python tests/test_low_latency.py --allow-mnvl

System Requirements and Validation

For DeepEP to actually utilize MNNVL, your NVSHMEM installation must be compiled with Multi-Node NVLink support. Without this build configuration, setting allow_mnnvl=True has no effect on communication paths.

The library includes validation logic in deep_ep/utils.py, where the check_nvlink_connections function inspects the GPU topology during buffer initialization. This ensures that the hardware is correctly detected before attempting MNNVL communication.

Performance Implications

When enabled on compatible hardware, MNNVL allows DeepEP to use NVLink for inter-node communication in addition to standard RDMA pathways. This configuration can significantly improve bandwidth and reduce latency for collective operations spanning multiple nodes, particularly for large-scale MoE model training where expert parallelism requires frequent cross-node transfers.

Summary

• DeepEP supports Multi-Node NVLink through the allow_mnnvl parameter in the Buffer class located in deep_ep/buffer.py.
• The default setting is False, which sets NVSHMEM_DISABLE_MNNVL=1 to disable MNNVL and ensure compatibility.
• Enable it by setting allow_mnnvl=True in Python or using --allow-mnnvl when running the test scripts.
• Successful operation requires NVSHMEM built with MNNVL support and appropriate hardware infrastructure detected by check_nvlink_connections in deep_ep/utils.py.

Frequently Asked Questions

What is Multi-Node NVLink (MNNVL) and why use it with DeepEP?

Multi-Node NVLink extends NVLink connectivity across server boundaries, allowing GPUs in different nodes to communicate directly at high bandwidth. Using MNNVL with DeepEP reduces latency and increases throughput for expert parallelism in distributed MoE training compared to pure InfiniBand or Ethernet RDMA solutions.

Is MNNVL enabled by default in DeepEP?

No. To prevent compatibility issues on systems without MNNVL-capable hardware or NVSHMEM builds, DeepEP explicitly disables MNNVL by default via the NVSHMEM_DISABLE_MNNVL environment variable. You must set allow_mnnvl=True (or pass --allow-mnnvl) to permit the NVSHMEM runtime to detect and use cross-node NVLink connections.

How can I verify my NVSHMEM installation supports MNNVL?

Check your NVSHMEM build configuration for MNNVL support flags, typically indicated during the configure or CMake stage. You can also inspect the deep_ep/utils.py file, which contains the check_nvlink_connections helper used during Buffer initialization to validate the NVLink topology across the allocated nodes.

Can I use MNNVL with DeepEP's low-latency kernels?

Yes. The tests/test_low_latency.py script accepts the --allow-mnnvl flag at line 322, confirming that low-latency communication kernels can operate over MNNVL when the feature is enabled. This allows high-frequency communication patterns to benefit from the lower latency of NVLink compared to traditional RDMA paths.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}