DeepSeek-V3 quantization_config Parameters Explained: e4m3, weight_block_size, and activation_scheme

The quantization_config in DeepSeek-V3 specifies the FP8 weight storage format, block-wise quantization granularity, and runtime activation scaling strategy required for efficient inference.

DeepSeek-V3 stores model weights in FP8 format to minimize memory usage while preserving accuracy. The quantization_config object within config.json (or embedded directly in weight files) tells the inference engine exactly how to interpret these compressed weights and how to quantize activation tensors during the forward pass. These parameters ensure compatibility between the saved checkpoint and the kernel implementations in the inference pipeline.

The fmt Parameter: FP8 Storage Format (e4m3)

The fmt field defines the specific FP8 datatype used for weight storage. When set to "e4m3", it maps directly to PyTorch's torch.float8_e4m3fn dtype, which allocates 4 bits to the exponent and 3 bits to the mantissa.

In inference/kernel.py, the act_quant function creates output tensors using this exact dtype:

y = torch.empty_like(x, dtype=torch.float8_e4m3fn)

The model architecture respects this setting in inference/model.py, where the Linear class assigns dtype = torch.float8_e4m3fn when building layers. This ensures all FP8 operations use the consistent e4m3 representation throughout the inference graph.

The weight_block_size Parameter: Quantization Granularity

The weight_block_size parameter (typically [128, 128]) determines the block-wise decomposition of weight matrices. Rather than applying a single scale factor to an entire layer, DeepSeek-V3 splits weights into 128×128 blocks, with each block maintaining its own inverse scale factor (weight_scale_inv).

During de-quantization in inference/kernel.py, the weight_dequant_kernel function processes these blocks individually. The scale tensor s has shape (M//block, N//block), and the kernel performs element-wise multiplication (y = x * s) to reconstruct full-precision weights. This granular approach minimizes quantization error across large matrices. The default block size of 128 is also hard-coded in the act_quant helper function at line 38 of the same file.

The activation_scheme Parameter: Runtime Scaling Strategy

The activation_scheme controls how activation tensors are quantized at inference time. The default "dynamic" scheme computes per-token (per-block) scaling factors on-the-fly rather than using pre-computed static values.

In inference/kernel.py, the dynamic quantization logic calculates the absolute maximum of each block, clamps it to a minimum of 1e-4, and converts it to a scale factor using the formula s = amax / 448.:

# From inference/kernel.py lines 24-33

amax = x.abs().max(dim=-1, keepdim=True)[0]
s = amax / 448.  # 448 is the max representable value in e4m3

The optional scale_fmt parameter allows reshaping these scales for specific kernel layouts, though the dynamic computation happens automatically during the forward pass. While other schemes (e.g., static) are reserved for future extensions, the current implementation exclusively supports dynamic activation quantization.

Practical Implementation Examples

Loading the Configuration from config.json

Access the quantization parameters before loading model weights:

import json
from pathlib import Path

config_path = Path("path/to/DeepSeek-V3/config.json")
with config_path.open() as f:
    cfg = json.load(f)

qcfg = cfg["quantization_config"]
fmt = qcfg["fmt"]                     # "e4m3"

weight_block = qcfg["weight_block_size"]  # [128, 128]

act_scheme = qcfg["activation_scheme"]   # "dynamic"

Quantizing Activations with Dynamic Scaling

Use the act_quant kernel with the block size from your configuration:

import torch
from inference.kernel import act_quant

x = torch.randn(1, 1024, device="cuda")   # example activation tensor

q_x, scale = act_quant(
    x, 
    block_size=weight_block[0], 
    scale_fmt=None
)

# q_x is now torch.float8_e4m3fn, scale contains per-block scaling factors

De-quantizing Weight Blocks

Convert FP8 weights back to full-precision (FP16/BF16) for inspection or mixed-precision operations:

import torch
from inference.kernel import weight_dequant

# Load FP8 weights and their corresponding scales

w_fp8 = torch.load("model0-w_fp8.safetensors")["weight"]      # float8_e4m3fn

w_scale = torch.load("model0-w_scale.safetensors")["scale"]  # float32

# De-quantize using the configured block size

w_full = weight_dequant(w_fp8, w_scale, block_size=128)

Integrating with Linear Layers

The Linear module in inference/model.py handles FP8 weights transparently:

from inference.model import Linear

# Initialize layer with FP8 dtype awareness

lin = Linear(in_features=1024, out_features=4096, dtype="fp8")
lin.weight.data = w_fp8          # Assign FP8 weight tensor

lin.weight_scale = w_scale       # Assign block-wise scales

# Forward pass automatically manages quantization/de-quantization

output = lin(x)

Summary

• fmt: Specifies the FP8 datatype (e4m3 maps to torch.float8_e4m3fn) used for all weight storage and activation quantization in the inference kernels.
• weight_block_size: Defines the 128×128 block granularity for weight quantization, enabling fine-grained scale factors that reduce quantization error compared to per-tensor scaling.
• activation_scheme: Controls runtime activation quantization; "dynamic" computes per-block scales on-the-fly using amax / 448. in the act_quant kernel.
• Configuration location: These parameters reside in config.json as documented in README_WEIGHTS.md, ensuring the inference engine matches the checkpoint's quantization strategy.

Frequently Asked Questions

What does the e4m3 format mean in DeepSeek-V3 quantization?

The e4m3 format is a specific FP8 representation with 4 exponent bits and 3 mantissa bits, implemented in PyTorch as torch.float8_e4m3fn. DeepSeek-V3 uses this dtype exclusively for storing compressed weights and intermediate activations, as defined by the fmt parameter in quantization_config. The act_quant kernel in inference/kernel.py explicitly creates tensors with this dtype when quantizing activations.

How does weight_block_size affect model performance?

The weight_block_size (default [128, 128]) balances memory efficiency against computational overhead. Smaller blocks provide finer granularity for scale factors, improving accuracy, but require storing more scale parameters (shape (M//128, N//128)). During inference, the weight_dequant_kernel multiplies each 128×128 block by its corresponding scale factor to reconstruct high-precision weights for matrix multiplication.

What is the difference between dynamic and static activation schemes?

Currently, DeepSeek-V3 only implements the dynamic activation scheme. This approach calculates scaling factors in real-time during the forward pass by finding the absolute maximum value in each block (amax), clamping it to 1e-4, and dividing by 448 (the maximum e4m3 value). A static scheme would use pre-computed scales, but the dynamic method adapts to varying input distributions without requiring calibration data.

Where is the quantization_config stored in DeepSeek-V3 files?

The quantization_config object is located in the repository's config.json file (or equivalent configuration files like config_v3.1.json in inference/configs/). According to README_WEIGHTS.md, this JSON schema documents the FP8 format, block size, and activation scheme, ensuring the inference kernels in inference/kernel.py and model definitions in inference/model.py correctly interpret the quantized checkpoint files.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}