DeepSeek-V3 671B Model Parallelism Requirements: Tensor Parallel Size Setup

To load the 671B-parameter DeepSeek-V3 checkpoint, you must configure a tensor-parallel size of 16, distributing the model across 16 GPUs using the --model-parallel flag during conversion and matching this sharding during inference.

The deepseek-ai/DeepSeek-V3 repository provides inference code for one of the largest open-source language models available. With 671 billion parameters, specific model parallelism requirements must be met to successfully deploy the model, as the checkpoint must be sharded using tensor parallelism to fit across multiple accelerator devices.

Why Tensor Parallelism Is Required for 671B Parameters

The sheer scale of DeepSeek-V3 necessitates model parallelism to distribute computational and memory workloads across hardware. For the 671B variant, the reference implementation mandates a specific tensor-parallel size of 16, meaning each GPU holds exactly 1/16th of the model weights.

This requirement is enforced in the checkpoint conversion pipeline. In inference/convert.py, the script defines a required command-line argument --model-parallel that specifies how many tensor-parallel shards to create when converting the Hugging Face checkpoint (lines 33-44). The sharding strategy partitions the model's parameters into separate safe tensors that must be loaded collectively during distributed inference.

Configuring Tensor Parallel Size During Conversion

Before running inference, you must convert the original checkpoint using the conversion script with the appropriate parallelism settings. The --model-parallel parameter dictates how many shards the 671B model will be split into.

According to the README documentation, the concrete example for the 671B model uses --model-parallel 16 alongside --n-experts 256 (line 288). This creates 16 separate shard files that must be distributed across your GPU cluster during inference.

python inference/convert.py \
    --hf-ckpt-path /path/to/hf_ckpt \
    --save-path /path/to/DeepSeek-V3-Demo \
    --n-experts 256 \
    --model-parallel 16

Deploying Inference with Matched Parallelism

The tensor-parallel size specified during conversion must exactly match the number of GPU processes launched during inference. For the 671B model, this means deploying across exactly 16 GPUs.

The README demonstrates this configuration using torchrun with 2 nodes and 8 GPUs per node, creating 16 total processes (lines 296-298). This 2×8 configuration aligns perfectly with the --model-parallel 16 setting used in the conversion step.

torchrun --nnodes 2 --nproc-per-node 8 \
    --node-rank $RANK --master-addr $MASTER_ADDR \
    inference/generate.py \
    --ckpt-path /path/to/DeepSeek-V3-Demo \
    --config inference/configs/config_671B.json \
    --interactive --temperature 0.7 --max-new-tokens 200

Configuration Files for 671B Deployment

The model architecture definition for the 671B parameter variant is stored in inference/configs/config_671B.json. This configuration file works in conjunction with the 16-way sharded checkpoints produced by the conversion script, ensuring that the inference engine correctly interprets the partitioned weight tensors across the distributed GPU mesh.

Summary

• The 671B-parameter DeepSeek-V3 requires a tensor-parallel size of 16 according to the reference implementation in deepseek-ai/DeepSeek-V3.
• Use the --model-parallel 16 argument in inference/convert.py (lines 33-44) to shard the checkpoint into 16 partitions during preprocessing.
• Deploy inference across exactly 16 GPUs (e.g., 2 nodes × 8 GPUs) using torchrun to match the conversion-time sharding strategy.
• The config_671B.json file provides the architectural configuration required to load these sharded weights correctly across the tensor-parallel topology.

Frequently Asked Questions

What is the exact tensor-parallel size required for DeepSeek-V3 671B?

The reference implementation requires a tensor-parallel size of 16. This value is specified in the example commands within the README (line 288) and enforced by the --model-parallel argument in inference/convert.py (lines 33-44).

How many GPUs are needed to run the 671B model with tensor parallelism?

You need 16 GPUs total, typically distributed as 2 nodes with 8 GPUs per node (or any topology summing to 16). This matches the 16-way sharding created during checkpoint conversion as shown in the README inference examples (lines 296-298).

Where is the tensor-parallel size configured in the DeepSeek-V3 codebase?

The tensor-parallel size is configured in inference/convert.py via the --model-parallel command-line argument. The same value must be reflected in your inference launch configuration using torchrun with --nproc-per-node and --nnodes settings that multiply to 16.

Can I use a different tensor-parallel size than 16 for the 671B model?

The reference implementation expects exactly 16 shards for the 671B model. Using a different tensor-parallel size would require modifying the conversion logic in convert.py and ensuring the inference engine in generate.py can handle alternative sharding schemes, which is not supported by the default config_671B.json configuration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}