How to Configure Multi-Node Tensor Parallelism for DeepSeek-V3 Inference in SGLang

Configure multi-node tensor parallelism for DeepSeek-V3 by launching SGLang v0.4.1+ with --tensor-parallel-size set to the total GPU count across nodes, using torchrun environment variables for rank coordination.

DeepSeek-V3 achieves state-of-the-art inference performance when served with SGLang, an open-source serving framework optimized for large language models. Starting with SGLang v0.4, the framework natively supports multi-node tensor parallelism, enabling you to shard the DeepSeek-V3 model across multiple physical machines connected via high-speed network interfaces.

Understanding Multi-Node Tensor Parallelism in SGLang

Multi-node tensor parallelism distributes individual transformer layers across GPUs located on different nodes. In the DeepSeek-V3 architecture, SGLang partitions the hidden dimension of weight matrices, with each node responsible for computing its slice of the forward pass.

Between layers, SGLang utilizes NCCL (or AMD-compatible equivalents) to perform collective communication operations, exchanging activation tensors across the network. This implementation follows the classic tensor-parallel pattern established by Megatron-LM and DeepSpeed, where process coordination occurs through a launcher service—either torchrun or SGLang's native launcher—that synchronizes ranks and establishes the global world size.

According to the DeepSeek-V3 repository, this integration requires no modifications to the model source code; the parallelism is handled entirely by SGLang's launch infrastructure, as referenced in the README at line 311.

Prerequisites for DeepSeek-V3 Multi-Node Deployment

Before configuring multi-node inference, verify your environment meets the following requirements:

• SGLang v0.4.1 or later: This version introduces full support for both NVIDIA and AMD GPUs in multi-node configurations, as documented in the DeepSeek-V3 README (lines 307-313).
• Network infrastructure: High-bandwidth, low-latency interconnects (InfiniBand or 100GbE+) between nodes to minimize communication overhead during tensor exchanges.
• Identical environments: All nodes must run the same SGLang version, CUDA/ROCm drivers, and DeepSeek-V3 model weights.

Launching DeepSeek-V3 with Multi-Node Tensor Parallelism

Environment Setup

Set the following environment variables on each node to enable process discovery:

export NODE_RANK=0          # Set to 0 on first node, 1 on second, etc.

export MASTER_ADDR=10.0.0.1 # IP address of the master node (rank 0)

export MASTER_PORT=29500    # Free port for coordination traffic

Launch Command Configuration

Execute the following command on every node, adjusting NODE_RANK per machine. The example below configures a 2-node deployment with 4 GPUs per node (8 total GPUs):

sglang serve \
    --model-path /path/to/deepseek-v3 \
    --dtype bf16 \
    --nproc_per_node 4 \
    --nnodes 2 \
    --node_rank $NODE_RANK \
    --master_addr $MASTER_ADDR \
    --master_port $MASTER_PORT \
    --tensor-parallel-size 8 \
    --port 8000

Key parameters explained:

• --tensor-parallel-size: Total number of GPUs across all nodes (must equal nproc_per_node × nnodes).
• --nproc_per_node: GPUs available on the current machine.
• --nnodes: Total number of physical machines in the cluster.

The same command runs on both physical machines; only the NODE_RANK environment variable differs between them.

Client Integration and Inference

Once the multi-node cluster is active, interact with the model using SGLang's HTTP endpoint:

import requests

url = "http://10.0.0.1:8000/generate"
payload = {
    "prompt": "Explain the benefits of tensor parallelism in large language model inference.",
    "max_new_tokens": 128,
    "temperature": 0.7,
}

response = requests.post(url, json=payload)
print(response.json()["generated_text"])

The client communicates only with the master node (rank 0), which coordinates the broadcast of input tensors to all tensor-parallel ranks and the reduction of partial computations back to the final output.

Optimizing Performance with FP8 and AMD GPUs

FP8 Quantization

For higher throughput on supported hardware, enable FP8 weight quantization by modifying the dtype parameter:

--dtype fp8   # Enables W8A8 inference for reduced memory bandwidth

This configuration reduces inter-node communication volume by transmitting 8-bit activations instead of 16-bit, significantly improving performance on bandwidth-constrained network interfaces.

AMD GPU Support

SGLang v0.4.1 includes native AMD ROCm kernels. To deploy on AMD hardware:

1. Set the appropriate device visibility variables:

   export HIP_VISIBLE_DEVICES=0,1,2,3

2. Use the identical launch command structure as NVIDIA deployments—the --tensor-parallel-size and node configuration parameters remain the same.

Summary

• Multi-node tensor parallelism in SGLang v0.4.1+ enables DeepSeek-V3 inference across multiple physical machines by sharding transformer layers and using NCCL for inter-node communication.
• Configuration requires setting NODE_RANK, MASTER_ADDR, and MASTER_PORT environment variables, then launching with --tensor-parallel-size equal to the total GPU count across all nodes.
• No code modifications are needed in the DeepSeek-V3 repository (inference/model.py or inference/generate.py); SGLang handles all parallelism logic externally.
• Performance optimizations include FP8 quantization (--dtype fp8) for reduced bandwidth and native AMD GPU support via ROCm kernels.

Frequently Asked Questions

How does multi-node tensor parallelism differ from single-node multi-GPU inference?

Single-node tensor parallelism uses high-speed NVLink or PCIe to communicate between GPUs within one server. Multi-node tensor parallelism extends this pattern across network interfaces (InfiniBand or Ethernet), allowing you to scale beyond the GPU capacity of a single machine. The DeepSeek-V3 model architecture remains identical; only the communication backend and launcher configuration change when moving from single-node to multi-node deployments.

What network bandwidth is required for efficient multi-node DeepSeek-V3 inference?

DeepSeek-V3's large hidden dimensions generate significant activation traffic between tensor-parallel ranks. For optimal performance, use InfiniBand HDR (200 Gbps) or faster links between nodes. If using Ethernet, ensure 100 Gbps+ with RDMA support (RoCE) to minimize latency during NCCL all-reduce operations. The FP8 quantization option (--dtype fp8) can halve the required bandwidth by transmitting 8-bit activations instead of bfloat16.

Can I mix NVIDIA and AMD GPUs in the same tensor-parallel group?

No, homogeneous hardware is required within a tensor-parallel group. All nodes must use identical GPU architectures (all NVIDIA or all AMD) because NCCL collectives and kernel implementations are architecture-specific. However, you can deploy separate SGLang clusters on different hardware types and route traffic to them using a load balancer. The DeepSeek-V3 weights are compatible with both NVIDIA (CUDA) and AMD (ROCm) implementations of SGLang.

Where are the model weights and configuration files located in the DeepSeek-V3 repository?

The inference configuration examples reside in inference/configs/config_16B.json, which defines model architecture parameters such as hidden size and number of layers. The core model implementation is in inference/model.py, which SGLang imports when you specify --model-path. The inference/generate.py script provides a single-node reference implementation, while inference/requirements.txt lists dependencies. For multi-node deployments, you do not execute these files directly; instead, point SGLang to the model path and it loads inference/model.py automatically.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}