# DeepSeek-V3 quantization_config Parameters Explained: e4m3, weight_block_size, and activation_scheme

> Understand DeepSeek-V3 quantization_config parameters like e4m3, weight_block_size, and activation_scheme to optimize LLM inference. Learn FP8 storage, block quantization, and activation scaling.

- Repository: [DeepSeek/DeepSeek-V3](https://github.com/deepseek-ai/DeepSeek-V3)
- Tags: deep-dive
- Published: 2026-02-26

---

**The `quantization_config` in DeepSeek-V3 specifies the FP8 weight storage format, block-wise quantization granularity, and runtime activation scaling strategy required for efficient inference.**

DeepSeek-V3 stores model weights in **FP8 format** to minimize memory usage while preserving accuracy. The `quantization_config` object within [`config.json`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/config.json) (or embedded directly in weight files) tells the inference engine exactly how to interpret these compressed weights and how to quantize activation tensors during the forward pass. These parameters ensure compatibility between the saved checkpoint and the kernel implementations in the inference pipeline.

## The `fmt` Parameter: FP8 Storage Format (e4m3)

The **`fmt`** field defines the specific FP8 datatype used for weight storage. When set to **`"e4m3"`**, it maps directly to PyTorch's `torch.float8_e4m3fn` dtype, which allocates 4 bits to the exponent and 3 bits to the mantissa.

In [`inference/kernel.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/kernel.py), the `act_quant` function creates output tensors using this exact dtype:

```python
y = torch.empty_like(x, dtype=torch.float8_e4m3fn)

```

The model architecture respects this setting in [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py), where the `Linear` class assigns `dtype = torch.float8_e4m3fn` when building layers. This ensures all FP8 operations use the consistent e4m3 representation throughout the inference graph.

## The `weight_block_size` Parameter: Quantization Granularity

The **`weight_block_size`** parameter (typically `[128, 128]`) determines the block-wise decomposition of weight matrices. Rather than applying a single scale factor to an entire layer, DeepSeek-V3 splits weights into 128×128 blocks, with each block maintaining its own inverse scale factor (`weight_scale_inv`).

During de-quantization in [`inference/kernel.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/kernel.py), the `weight_dequant_kernel` function processes these blocks individually. The scale tensor `s` has shape `(M//block, N//block)`, and the kernel performs element-wise multiplication (`y = x * s`) to reconstruct full-precision weights. This granular approach minimizes quantization error across large matrices. The default block size of 128 is also hard-coded in the `act_quant` helper function at line 38 of the same file.

## The `activation_scheme` Parameter: Runtime Scaling Strategy

The **`activation_scheme`** controls how activation tensors are quantized at inference time. The default **`"dynamic"`** scheme computes per-token (per-block) scaling factors on-the-fly rather than using pre-computed static values.

In [`inference/kernel.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/kernel.py), the dynamic quantization logic calculates the absolute maximum of each block, clamps it to a minimum of `1e-4`, and converts it to a scale factor using the formula `s = amax / 448.`:

```python

# From inference/kernel.py lines 24-33

amax = x.abs().max(dim=-1, keepdim=True)[0]
s = amax / 448.  # 448 is the max representable value in e4m3

```

The optional `scale_fmt` parameter allows reshaping these scales for specific kernel layouts, though the dynamic computation happens automatically during the forward pass. While other schemes (e.g., static) are reserved for future extensions, the current implementation exclusively supports dynamic activation quantization.

## Practical Implementation Examples

### Loading the Configuration from config.json

Access the quantization parameters before loading model weights:

```python
import json
from pathlib import Path

config_path = Path("path/to/DeepSeek-V3/config.json")
with config_path.open() as f:
    cfg = json.load(f)

qcfg = cfg["quantization_config"]
fmt = qcfg["fmt"]                     # "e4m3"

weight_block = qcfg["weight_block_size"]  # [128, 128]

act_scheme = qcfg["activation_scheme"]   # "dynamic"

```

### Quantizing Activations with Dynamic Scaling

Use the `act_quant` kernel with the block size from your configuration:

```python
import torch
from inference.kernel import act_quant

x = torch.randn(1, 1024, device="cuda")   # example activation tensor

q_x, scale = act_quant(
    x, 
    block_size=weight_block[0], 
    scale_fmt=None
)

# q_x is now torch.float8_e4m3fn, scale contains per-block scaling factors

```

### De-quantizing Weight Blocks

Convert FP8 weights back to full-precision (FP16/BF16) for inspection or mixed-precision operations:

```python
import torch
from inference.kernel import weight_dequant

# Load FP8 weights and their corresponding scales

w_fp8 = torch.load("model0-w_fp8.safetensors")["weight"]      # float8_e4m3fn

w_scale = torch.load("model0-w_scale.safetensors")["scale"]  # float32

# De-quantize using the configured block size

w_full = weight_dequant(w_fp8, w_scale, block_size=128)

```

### Integrating with Linear Layers

The `Linear` module in [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py) handles FP8 weights transparently:

```python
from inference.model import Linear

# Initialize layer with FP8 dtype awareness

lin = Linear(in_features=1024, out_features=4096, dtype="fp8")
lin.weight.data = w_fp8          # Assign FP8 weight tensor

lin.weight_scale = w_scale       # Assign block-wise scales

# Forward pass automatically manages quantization/de-quantization

output = lin(x)

```

## Summary

- **`fmt`**: Specifies the FP8 datatype (`e4m3` maps to `torch.float8_e4m3fn`) used for all weight storage and activation quantization in the inference kernels.
- **`weight_block_size`**: Defines the 128×128 block granularity for weight quantization, enabling fine-grained scale factors that reduce quantization error compared to per-tensor scaling.
- **`activation_scheme`**: Controls runtime activation quantization; `"dynamic"` computes per-block scales on-the-fly using `amax / 448.` in the `act_quant` kernel.
- **Configuration location**: These parameters reside in [`config.json`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/config.json) as documented in [`README_WEIGHTS.md`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/README_WEIGHTS.md), ensuring the inference engine matches the checkpoint's quantization strategy.

## Frequently Asked Questions

### What does the e4m3 format mean in DeepSeek-V3 quantization?

The **e4m3** format is a specific FP8 representation with 4 exponent bits and 3 mantissa bits, implemented in PyTorch as `torch.float8_e4m3fn`. DeepSeek-V3 uses this dtype exclusively for storing compressed weights and intermediate activations, as defined by the `fmt` parameter in `quantization_config`. The `act_quant` kernel in [`inference/kernel.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/kernel.py) explicitly creates tensors with this dtype when quantizing activations.

### How does weight_block_size affect model performance?

The **weight_block_size** (default `[128, 128]`) balances memory efficiency against computational overhead. Smaller blocks provide finer granularity for scale factors, improving accuracy, but require storing more scale parameters (shape `(M//128, N//128)`). During inference, the `weight_dequant_kernel` multiplies each 128×128 block by its corresponding scale factor to reconstruct high-precision weights for matrix multiplication.

### What is the difference between dynamic and static activation schemes?

Currently, DeepSeek-V3 only implements the **dynamic** activation scheme. This approach calculates scaling factors in real-time during the forward pass by finding the absolute maximum value in each block (`amax`), clamping it to `1e-4`, and dividing by 448 (the maximum e4m3 value). A static scheme would use pre-computed scales, but the dynamic method adapts to varying input distributions without requiring calibration data.

### Where is the quantization_config stored in DeepSeek-V3 files?

The `quantization_config` object is located in the repository's [`config.json`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/config.json) file (or equivalent configuration files like [`config_v3.1.json`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/config_v3.1.json) in `inference/configs/`). According to [`README_WEIGHTS.md`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/README_WEIGHTS.md), this JSON schema documents the FP8 format, block size, and activation scheme, ensuring the inference kernels in [`inference/kernel.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/kernel.py) and model definitions in [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py) correctly interpret the quantized checkpoint files.