# NCCL Configuration for Cross-Node Communication in DeepSeek-v4-Flash-DSpark

> Configure NCCL for cross-node communication with DeepSeek-v4-Flash-DSpark. Learn how environment variables bind ConnectX NICs for fast GPU tensor transfers.

- Repository: [Mia's AI Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark](https://github.com/MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark)
- Tags: how-to-guide
- Published: 2026-09-09

---

**The DeepSeek-v4-Flash-DSpark stack configures NCCL for cross-node communication through mandatory environment variables `NCCL_SOCKET_IFNAME` and `NCCL_IB_HCA`, which bind ConnectX RoCE NICs to enable high-throughput tensor transfers between GPU nodes.**

The MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark repository implements distributed inference across multiple GPU-equipped nodes using **NCCL** (NVIDIA Collective Communications Library) over RoCEv2. Understanding the NCCL configuration for cross-node communication is essential for deploying this multi-node vLLM stack, as the launch scripts strictly validate specific environment variables before initializing the high-speed fabric.

## Required Environment Variables

The configuration depends on two mandatory variables that the startup script validates before execution. In [`start-deepseek-v4-flash-dspark.sh`](https://github.com/MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark/blob/main/start-deepseek-v4-flash-dspark.sh) (lines 51-55), the script performs strict checks and aborts if either variable is missing:

```bash
: "${NCCL_IB_HCA:?NCCL_IB_HCA must be set in $ENV_FILE}"
: "${NCCL_SOCKET_IFNAME:?NCCL_SOCKET_IFNAME must be set in $ENV_FILE}"

```

**`NCCL_SOCKET_IFNAME`** specifies the network interface that NCCL binds to for socket-based bootstrap and control traffic. According to [`docs/ENVS.md`](https://github.com/MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark/blob/main/docs/ENVS.md) (line 81), the typical value for the ConnectX RoCE NIC is `enp1s0f1np1`.

**`NCCL_IB_HCA`** identifies the specific InfiniBand Host Channel Adapter. As referenced in [`scripts/test-dspark-api-keys.py`](https://github.com/MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark/blob/main/scripts/test-dspark-api-keys.py) (line 916), the repository uses `mlx5_0` to target the Mellanox ConnectX device.

## Optional NCCL Tuning Parameters

Beyond the mandatory interface bindings, the repository supports several advanced knobs for fine-tuning cross-node performance:

- **`NCCL_IB_GID_AUTO`**: When set to `1` (default), NCCL automatically discovers the RoCEv2 GID for the selected HCA. Set to `0` to manually pin a specific GID index.
- **`NCCL_IB_GID_INDEX`**: Specifies the GID index manually when `NCCL_IB_GID_AUTO=0`.
- **`NCCL_IB_MERGE_NICS`**: Controls dual-port NIC merging behavior. The repository leaves this unset, defaulting to NCCL's standard merging behavior (`1`).
- **`NCCL_NET_GDR_LEVEL`**, **`NCCL_NET_GDR_READ`**, **`NCCL_DMABUF_ENABLE`**, **`NCCL_GIN_ENABLE`**: Advanced GPU Direct RDMA parameters passed through unchanged when set, though they remain unset in the default configuration.

## Cross-Node Bootstrap and Validation Flow

The repository implements a four-phase validation sequence in [`start-deepseek-v4-flash-dspark.sh`](https://github.com/MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark/blob/main/start-deepseek-v4-flash-dspark.sh) (lines 1070-1085) to ensure reliable multi-node communication:

1. **Interface Selection**: The head node initializes `NCCL_SOCKET_IFNAME` (e.g., `enp1s0f1np1`).
2. **Worker Injection**: The script copies the head node value to `WORKER_NCCL_SOCKET_IFNAME` (or `WORKER2_NCCL_SOCKET_IFNAME` for TP = 3 configurations).
3. **GID Validation**: The `pick_gid_match_ip` function resolves the RoCEv2 IPv4 address for the selected HCA, confirming that head and worker nodes resolve to compatible GIDs.
4. **NCCL Initialization**: With validated environment variables present, PyTorch's `ProcessGroupNCCL` (used by vLLM) automatically discovers the NICs and establishes the high-throughput channel over RoCEv2.

## Docker Compose Environment Propagation

The launch scripts propagate NCCL configuration into containers through the generated [`docker-compose.dspark.yml`](https://github.com/MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark/blob/main/docker-compose.dspark.yml) file. The compose file injects the socket interface variables into the container environment, cascading the settings to related transport backends:

```yaml

# docker-compose.dspark.yml (excerpt)

environment:
  NCCL_SOCKET_IFNAME: "${NCCL_SOCKET_IFNAME}"
  TP_SOCKET_IFNAME: "${TP_SOCKET_IFNAME:-${NCCL_SOCKET_IFNAME}}"
  GLOO_SOCKET_IFNAME: "${GLOO_SOCKET_IFNAME:-${NCCL_SOCKET_IFNAME}}"

```

This ensures consistent network interface selection across NCCL, Tensor Parallelism (TP), and GLOO process groups.

## Example Configuration

The repository provides a concrete template in `.env.dspark.example` (lines 54-62):

```text
NCCL_SOCKET_IFNAME=enpXsYfZnpN
NCCL_IB_HCA=mlx5_0

```

Replace `enpXsYfZnpN` with your specific ConnectX interface name (e.g., `enp1s0f1np1`) to match your hardware configuration.

## Summary

- **`NCCL_SOCKET_IFNAME`** and **`NCCL_IB_HCA`** are mandatory variables validated by [`start-deepseek-v4-flash-dspark.sh`](https://github.com/MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark/blob/main/start-deepseek-v4-flash-dspark.sh) before container initialization.
- The `pick_gid_match_ip` function validates RoCEv2 GID compatibility between head and worker nodes during startup (lines 1070-1085).
- Docker Compose propagates interface settings to worker containers, ensuring consistent fabric configuration across the cluster.
- The default configuration targets Mellanox ConnectX adapters (`mlx5_0`) using RoCEv2 for low-latency, high-bandwidth tensor transfers.

## Frequently Asked Questions

### What happens if NCCL_SOCKET_IFNAME is not set?

The startup script will abort immediately with an error message. Specifically, the parameter expansion `${NCCL_SOCKET_IFNAME:?...}` in [`start-deepseek-v4-flash-dspark.sh`](https://github.com/MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark/blob/main/start-deepseek-v4-flash-dspark.sh) (line 52) causes the script to exit and report that the variable must be defined in the environment file.

### Why is mlx5_0 used as the default NCCL_IB_HCA?

The value `mlx5_0` corresponds to the Mellanox ConnectX adapter present in the reference DGX infrastructure. As documented in [`scripts/test-dspark-api-keys.py`](https://github.com/MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark/blob/main/scripts/test-dspark-api-keys.py) (line 916), this HCA device name provides the RoCEv2 capability required for cross-node GPU communication.

### How does the system validate GID compatibility between nodes?

The `pick_gid_match_ip` function resolves the RoCEv2 IPv4 addresses for the selected HCAs on both head and worker nodes. During the validation phase (lines 1070-1085 of the start script), it confirms that the nodes resolve to compatible GID indexes before allowing NCCL initialization to proceed.

### Can I disable dual-port NIC merging in this configuration?

Yes. While the repository leaves `NCCL_IB_MERGE_NICS` unset (defaulting to `1`), you can explicitly set `NCCL_IB_MERGE_NICS=0` in your environment file to prevent NCCL from merging dual-port ConnectX adapters, which may be necessary for specific network topologies.