# DeepSeek-V3 671B Model Parallelism Requirements: Tensor Parallel Size Setup

> Learn the tensor parallel size for DeepSeek-V3 671B model loading. Discover how to set up model parallelism across 16 GPUs for efficient inference.

- Repository: [DeepSeek/DeepSeek-V3](https://github.com/deepseek-ai/DeepSeek-V3)
- Tags: internals
- Published: 2026-02-26

---

**To load the 671B-parameter DeepSeek-V3 checkpoint, you must configure a tensor-parallel size of 16, distributing the model across 16 GPUs using the `--model-parallel` flag during conversion and matching this sharding during inference.**

The deepseek-ai/DeepSeek-V3 repository provides inference code for one of the largest open-source language models available. With 671 billion parameters, specific model parallelism requirements must be met to successfully deploy the model, as the checkpoint must be sharded using tensor parallelism to fit across multiple accelerator devices.

## Why Tensor Parallelism Is Required for 671B Parameters

The sheer scale of DeepSeek-V3 necessitates **model parallelism** to distribute computational and memory workloads across hardware. For the 671B variant, the reference implementation mandates a specific **tensor-parallel size** of 16, meaning each GPU holds exactly 1/16th of the model weights.

This requirement is enforced in the checkpoint conversion pipeline. In [`inference/convert.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/convert.py), the script defines a required command-line argument `--model-parallel` that specifies how many tensor-parallel shards to create when converting the Hugging Face checkpoint (lines 33-44). The sharding strategy partitions the model's parameters into separate safe tensors that must be loaded collectively during distributed inference.

## Configuring Tensor Parallel Size During Conversion

Before running inference, you must convert the original checkpoint using the conversion script with the appropriate parallelism settings. The `--model-parallel` parameter dictates how many shards the 671B model will be split into.

According to the README documentation, the concrete example for the 671B model uses `--model-parallel 16` alongside `--n-experts 256` (line 288). This creates 16 separate shard files that must be distributed across your GPU cluster during inference.

```bash
python inference/convert.py \
    --hf-ckpt-path /path/to/hf_ckpt \
    --save-path /path/to/DeepSeek-V3-Demo \
    --n-experts 256 \
    --model-parallel 16

```

## Deploying Inference with Matched Parallelism

The tensor-parallel size specified during conversion must exactly match the number of GPU processes launched during inference. For the 671B model, this means deploying across exactly 16 GPUs.

The README demonstrates this configuration using `torchrun` with 2 nodes and 8 GPUs per node, creating 16 total processes (lines 296-298). This 2×8 configuration aligns perfectly with the `--model-parallel 16` setting used in the conversion step.

```bash
torchrun --nnodes 2 --nproc-per-node 8 \
    --node-rank $RANK --master-addr $MASTER_ADDR \
    inference/generate.py \
    --ckpt-path /path/to/DeepSeek-V3-Demo \
    --config inference/configs/config_671B.json \
    --interactive --temperature 0.7 --max-new-tokens 200

```

## Configuration Files for 671B Deployment

The model architecture definition for the 671B parameter variant is stored in [`inference/configs/config_671B.json`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/configs/config_671B.json). This configuration file works in conjunction with the 16-way sharded checkpoints produced by the conversion script, ensuring that the inference engine correctly interprets the partitioned weight tensors across the distributed GPU mesh.

## Summary

- The 671B-parameter DeepSeek-V3 requires a **tensor-parallel size of 16** according to the reference implementation in `deepseek-ai/DeepSeek-V3`.
- Use the `--model-parallel 16` argument in [`inference/convert.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/convert.py) (lines 33-44) to shard the checkpoint into 16 partitions during preprocessing.
- Deploy inference across exactly **16 GPUs** (e.g., 2 nodes × 8 GPUs) using `torchrun` to match the conversion-time sharding strategy.
- The [`config_671B.json`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/config_671B.json) file provides the architectural configuration required to load these sharded weights correctly across the tensor-parallel topology.

## Frequently Asked Questions

### What is the exact tensor-parallel size required for DeepSeek-V3 671B?

The reference implementation requires a **tensor-parallel size of 16**. This value is specified in the example commands within the README (line 288) and enforced by the `--model-parallel` argument in [`inference/convert.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/convert.py) (lines 33-44).

### How many GPUs are needed to run the 671B model with tensor parallelism?

You need **16 GPUs** total, typically distributed as 2 nodes with 8 GPUs per node (or any topology summing to 16). This matches the 16-way sharding created during checkpoint conversion as shown in the README inference examples (lines 296-298).

### Where is the tensor-parallel size configured in the DeepSeek-V3 codebase?

The tensor-parallel size is configured in [`inference/convert.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/convert.py) via the `--model-parallel` command-line argument. The same value must be reflected in your inference launch configuration using `torchrun` with `--nproc-per-node` and `--nnodes` settings that multiply to 16.

### Can I use a different tensor-parallel size than 16 for the 671B model?

The reference implementation expects exactly 16 shards for the 671B model. Using a different tensor-parallel size would require modifying the conversion logic in [`convert.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/convert.py) and ensuring the inference engine in [`generate.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/generate.py) can handle alternative sharding schemes, which is not supported by the default [`config_671B.json`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/config_671B.json) configuration.