NCCL Configuration for Cross-Node Communication in DeepSeek-v4-Flash-DSpark

The DeepSeek-v4-Flash-DSpark stack configures NCCL for cross-node communication through mandatory environment variables NCCL_SOCKET_IFNAME and NCCL_IB_HCA, which bind ConnectX RoCE NICs to enable high-throughput tensor transfers between GPU nodes.

The MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark repository implements distributed inference across multiple GPU-equipped nodes using NCCL (NVIDIA Collective Communications Library) over RoCEv2. Understanding the NCCL configuration for cross-node communication is essential for deploying this multi-node vLLM stack, as the launch scripts strictly validate specific environment variables before initializing the high-speed fabric.

Required Environment Variables

The configuration depends on two mandatory variables that the startup script validates before execution. In start-deepseek-v4-flash-dspark.sh (lines 51-55), the script performs strict checks and aborts if either variable is missing:

: "${NCCL_IB_HCA:?NCCL_IB_HCA must be set in $ENV_FILE}"
: "${NCCL_SOCKET_IFNAME:?NCCL_SOCKET_IFNAME must be set in $ENV_FILE}"

NCCL_SOCKET_IFNAME specifies the network interface that NCCL binds to for socket-based bootstrap and control traffic. According to docs/ENVS.md (line 81), the typical value for the ConnectX RoCE NIC is enp1s0f1np1.

NCCL_IB_HCA identifies the specific InfiniBand Host Channel Adapter. As referenced in scripts/test-dspark-api-keys.py (line 916), the repository uses mlx5_0 to target the Mellanox ConnectX device.

Optional NCCL Tuning Parameters

Beyond the mandatory interface bindings, the repository supports several advanced knobs for fine-tuning cross-node performance:

• NCCL_IB_GID_AUTO: When set to 1 (default), NCCL automatically discovers the RoCEv2 GID for the selected HCA. Set to 0 to manually pin a specific GID index.
• NCCL_IB_GID_INDEX: Specifies the GID index manually when NCCL_IB_GID_AUTO=0.
• NCCL_IB_MERGE_NICS: Controls dual-port NIC merging behavior. The repository leaves this unset, defaulting to NCCL's standard merging behavior (1).
• NCCL_NET_GDR_LEVEL, NCCL_NET_GDR_READ, NCCL_DMABUF_ENABLE, NCCL_GIN_ENABLE: Advanced GPU Direct RDMA parameters passed through unchanged when set, though they remain unset in the default configuration.

Cross-Node Bootstrap and Validation Flow

The repository implements a four-phase validation sequence in start-deepseek-v4-flash-dspark.sh (lines 1070-1085) to ensure reliable multi-node communication:

1. Interface Selection: The head node initializes NCCL_SOCKET_IFNAME (e.g., enp1s0f1np1).
2. Worker Injection: The script copies the head node value to WORKER_NCCL_SOCKET_IFNAME (or WORKER2_NCCL_SOCKET_IFNAME for TP = 3 configurations).
3. GID Validation: The pick_gid_match_ip function resolves the RoCEv2 IPv4 address for the selected HCA, confirming that head and worker nodes resolve to compatible GIDs.
4. NCCL Initialization: With validated environment variables present, PyTorch's ProcessGroupNCCL (used by vLLM) automatically discovers the NICs and establishes the high-throughput channel over RoCEv2.

Docker Compose Environment Propagation

The launch scripts propagate NCCL configuration into containers through the generated docker-compose.dspark.yml file. The compose file injects the socket interface variables into the container environment, cascading the settings to related transport backends:

# docker-compose.dspark.yml (excerpt)

environment:
  NCCL_SOCKET_IFNAME: "${NCCL_SOCKET_IFNAME}"
  TP_SOCKET_IFNAME: "${TP_SOCKET_IFNAME:-${NCCL_SOCKET_IFNAME}}"
  GLOO_SOCKET_IFNAME: "${GLOO_SOCKET_IFNAME:-${NCCL_SOCKET_IFNAME}}"

This ensures consistent network interface selection across NCCL, Tensor Parallelism (TP), and GLOO process groups.

Example Configuration

The repository provides a concrete template in .env.dspark.example (lines 54-62):

NCCL_SOCKET_IFNAME=enpXsYfZnpN
NCCL_IB_HCA=mlx5_0

Replace enpXsYfZnpN with your specific ConnectX interface name (e.g., enp1s0f1np1) to match your hardware configuration.

Summary

• NCCL_SOCKET_IFNAME and NCCL_IB_HCA are mandatory variables validated by start-deepseek-v4-flash-dspark.sh before container initialization.
• The pick_gid_match_ip function validates RoCEv2 GID compatibility between head and worker nodes during startup (lines 1070-1085).
• Docker Compose propagates interface settings to worker containers, ensuring consistent fabric configuration across the cluster.
• The default configuration targets Mellanox ConnectX adapters (mlx5_0) using RoCEv2 for low-latency, high-bandwidth tensor transfers.

Frequently Asked Questions

What happens if NCCL_SOCKET_IFNAME is not set?

The startup script will abort immediately with an error message. Specifically, the parameter expansion ${NCCL_SOCKET_IFNAME:?...} in start-deepseek-v4-flash-dspark.sh (line 52) causes the script to exit and report that the variable must be defined in the environment file.

Why is mlx5_0 used as the default NCCL_IB_HCA?

The value mlx5_0 corresponds to the Mellanox ConnectX adapter present in the reference DGX infrastructure. As documented in scripts/test-dspark-api-keys.py (line 916), this HCA device name provides the RoCEv2 capability required for cross-node GPU communication.

How does the system validate GID compatibility between nodes?

The pick_gid_match_ip function resolves the RoCEv2 IPv4 addresses for the selected HCAs on both head and worker nodes. During the validation phase (lines 1070-1085 of the start script), it confirms that the nodes resolve to compatible GID indexes before allowing NCCL initialization to proceed.

Can I disable dual-port NIC merging in this configuration?

Yes. While the repository leaves NCCL_IB_MERGE_NICS unset (defaulting to 1), you can explicitly set NCCL_IB_MERGE_NICS=0 in your environment file to prevent NCCL from merging dual-port ConnectX adapters, which may be necessary for specific network topologies.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}