# How to Configure Multi-Node Tensor Parallelism for DeepSeek-V3 Inference in SGLang

> Learn to configure multi-node tensor parallelism for DeepSeek-V3 inference using SGLang. Easily scale your DeepSeek-V3 models across multiple machines with our guide.

- Repository: [DeepSeek/DeepSeek-V3](https://github.com/deepseek-ai/DeepSeek-V3)
- Tags: how-to-guide
- Published: 2026-02-26

---

**Configure multi-node tensor parallelism for DeepSeek-V3 by launching SGLang v0.4.1+ with `--tensor-parallel-size` set to the total GPU count across nodes, using `torchrun` environment variables for rank coordination.**

DeepSeek-V3 achieves state-of-the-art inference performance when served with **SGLang**, an open-source serving framework optimized for large language models. Starting with SGLang v0.4, the framework natively supports **multi-node tensor parallelism**, enabling you to shard the DeepSeek-V3 model across multiple physical machines connected via high-speed network interfaces.

## Understanding Multi-Node Tensor Parallelism in SGLang

Multi-node tensor parallelism distributes individual transformer layers across GPUs located on different nodes. In the DeepSeek-V3 architecture, SGLang partitions the hidden dimension of weight matrices, with each node responsible for computing its slice of the forward pass.

Between layers, SGLang utilizes **NCCL** (or AMD-compatible equivalents) to perform collective communication operations, exchanging activation tensors across the network. This implementation follows the classic tensor-parallel pattern established by Megatron-LM and DeepSpeed, where process coordination occurs through a launcher service—either `torchrun` or SGLang's native launcher—that synchronizes ranks and establishes the global world size.

According to the DeepSeek-V3 repository, this integration requires no modifications to the model source code; the parallelism is handled entirely by SGLang's launch infrastructure, as referenced in the README at line 311.

## Prerequisites for DeepSeek-V3 Multi-Node Deployment

Before configuring multi-node inference, verify your environment meets the following requirements:

- **SGLang v0.4.1 or later**: This version introduces full support for both NVIDIA and AMD GPUs in multi-node configurations, as documented in the DeepSeek-V3 README (lines 307-313).
- **Network infrastructure**: High-bandwidth, low-latency interconnects (InfiniBand or 100GbE+) between nodes to minimize communication overhead during tensor exchanges.
- **Identical environments**: All nodes must run the same SGLang version, CUDA/ROCm drivers, and DeepSeek-V3 model weights.

## Launching DeepSeek-V3 with Multi-Node Tensor Parallelism

### Environment Setup

Set the following environment variables on each node to enable process discovery:

```bash
export NODE_RANK=0          # Set to 0 on first node, 1 on second, etc.

export MASTER_ADDR=10.0.0.1 # IP address of the master node (rank 0)

export MASTER_PORT=29500    # Free port for coordination traffic

```

### Launch Command Configuration

Execute the following command on every node, adjusting `NODE_RANK` per machine. The example below configures a 2-node deployment with 4 GPUs per node (8 total GPUs):

```bash
sglang serve \
    --model-path /path/to/deepseek-v3 \
    --dtype bf16 \
    --nproc_per_node 4 \
    --nnodes 2 \
    --node_rank $NODE_RANK \
    --master_addr $MASTER_ADDR \
    --master_port $MASTER_PORT \
    --tensor-parallel-size 8 \
    --port 8000

```

**Key parameters explained:**

- `--tensor-parallel-size`: Total number of GPUs across all nodes (must equal `nproc_per_node × nnodes`).
- `--nproc_per_node`: GPUs available on the current machine.
- `--nnodes`: Total number of physical machines in the cluster.

The same command runs on both physical machines; only the `NODE_RANK` environment variable differs between them.

## Client Integration and Inference

Once the multi-node cluster is active, interact with the model using SGLang's HTTP endpoint:

```python
import requests

url = "http://10.0.0.1:8000/generate"
payload = {
    "prompt": "Explain the benefits of tensor parallelism in large language model inference.",
    "max_new_tokens": 128,
    "temperature": 0.7,
}

response = requests.post(url, json=payload)
print(response.json()["generated_text"])

```

The client communicates only with the master node (rank 0), which coordinates the broadcast of input tensors to all tensor-parallel ranks and the reduction of partial computations back to the final output.

## Optimizing Performance with FP8 and AMD GPUs

### FP8 Quantization

For higher throughput on supported hardware, enable FP8 weight quantization by modifying the dtype parameter:

```bash
--dtype fp8   # Enables W8A8 inference for reduced memory bandwidth

```

This configuration reduces inter-node communication volume by transmitting 8-bit activations instead of 16-bit, significantly improving performance on bandwidth-constrained network interfaces.

### AMD GPU Support

SGLang v0.4.1 includes native AMD ROCm kernels. To deploy on AMD hardware:

1. Set the appropriate device visibility variables:
   ```bash
   export HIP_VISIBLE_DEVICES=0,1,2,3
   ```

2. Use the identical launch command structure as NVIDIA deployments—the `--tensor-parallel-size` and node configuration parameters remain the same.

## Summary

- **Multi-node tensor parallelism** in SGLang v0.4.1+ enables DeepSeek-V3 inference across multiple physical machines by sharding transformer layers and using NCCL for inter-node communication.
- **Configuration requires** setting `NODE_RANK`, `MASTER_ADDR`, and `MASTER_PORT` environment variables, then launching with `--tensor-parallel-size` equal to the total GPU count across all nodes.
- **No code modifications** are needed in the DeepSeek-V3 repository ([`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py) or [`inference/generate.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/generate.py)); SGLang handles all parallelism logic externally.
- **Performance optimizations** include FP8 quantization (`--dtype fp8`) for reduced bandwidth and native AMD GPU support via ROCm kernels.

## Frequently Asked Questions

### How does multi-node tensor parallelism differ from single-node multi-GPU inference?

Single-node tensor parallelism uses high-speed NVLink or PCIe to communicate between GPUs within one server. Multi-node tensor parallelism extends this pattern across network interfaces (InfiniBand or Ethernet), allowing you to scale beyond the GPU capacity of a single machine. The DeepSeek-V3 model architecture remains identical; only the communication backend and launcher configuration change when moving from single-node to multi-node deployments.

### What network bandwidth is required for efficient multi-node DeepSeek-V3 inference?

DeepSeek-V3's large hidden dimensions generate significant activation traffic between tensor-parallel ranks. For optimal performance, use InfiniBand HDR (200 Gbps) or faster links between nodes. If using Ethernet, ensure 100 Gbps+ with RDMA support (RoCE) to minimize latency during NCCL all-reduce operations. The FP8 quantization option (`--dtype fp8`) can halve the required bandwidth by transmitting 8-bit activations instead of bfloat16.

### Can I mix NVIDIA and AMD GPUs in the same tensor-parallel group?

No, homogeneous hardware is required within a tensor-parallel group. All nodes must use identical GPU architectures (all NVIDIA or all AMD) because NCCL collectives and kernel implementations are architecture-specific. However, you can deploy separate SGLang clusters on different hardware types and route traffic to them using a load balancer. The DeepSeek-V3 weights are compatible with both NVIDIA (CUDA) and AMD (ROCm) implementations of SGLang.

### Where are the model weights and configuration files located in the DeepSeek-V3 repository?

The inference configuration examples reside in [`inference/configs/config_16B.json`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/configs/config_16B.json), which defines model architecture parameters such as hidden size and number of layers. The core model implementation is in [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py), which SGLang imports when you specify `--model-path`. The [`inference/generate.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/generate.py) script provides a single-node reference implementation, while [`inference/requirements.txt`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/requirements.txt) lists dependencies. For multi-node deployments, you do not execute these files directly; instead, point SGLang to the model path and it loads [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py) automatically.