Understanding Dynamic and Static Activation Quantization Schemes in DeepSeek-V3

Dynamic activation quantization computes scaling factors at runtime based on actual activation values, while static activation quantization uses pre-computed scales determined during model calibration. DeepSeek-V3 exclusively implements the dynamic scheme, calculating per-block scales on-the-fly during inference for maximum accuracy.

Activation quantization reduces intermediate tensor precision from full-precision formats like FP32 to low-bit representations such as FP8. The choice between dynamic and static activation quantization schemes fundamentally determines how scaling factors map floating-point values to quantized integers. According to the DeepSeek-V3 source code, the repository implements dynamic quantization through kernel-level functions that adapt to real-time input distributions.

How Dynamic Activation Quantization Works

Dynamic quantization calculates scaling factors for every activation block during inference based on the actual runtime data. In DeepSeek-V3, this is handled by the act_quant function in inference/kernel.py (lines 24-33), which processes contiguous tensors and derives scales from the maximum absolute values found in each block.

The implementation uses a default block size of 128 channels, computing the scale as s = amax / 448 where amax is the clamped maximum absolute value. When scale_fmt is set to "ue8m0", the function optionally rounds scales to power-of-two values for hardware compatibility. The linear wrapper in inference/model.py (lines 159-166) calls act_quant during forward passes, integrating dynamic quantization into the generation pipeline.

import torch
from inference.kernel import act_quant

# x: a contiguous tensor whose last dimension is a multiple of block_size (default 128)

x = torch.randn(1, 1024, dtype=torch.float32)

# Quantize activations on the fly

q_x, scale = act_quant(x, block_size=128, scale_fmt=None)

# q_x has dtype torch.float8_e4m3fn, scale holds per‑block FP32 scales

How Static Activation Quantization Differs

Static quantization relies on scales computed ahead of time, typically during model preparation or calibration on representative datasets. These pre-determined scales are stored alongside weights in the checkpoint files and applied uniformly at inference time, eliminating the need for per-input computation.

Unlike the dynamic approach, static schemes do not require the act_quant function to analyze activation statistics during forward passes. Instead, the model loads static_scales tensors and applies them directly to dequantize values. While this reduces runtime overhead, it cannot adapt when input distributions differ from the calibration data.

# Assume `static_scales` is a tensor saved with the model checkpoint

# and `static_dequant` simply divides by that scale without recomputing it.

def static_dequant(q_x, static_scales):
    return q_x.float() * static_scales.unsqueeze(-1)

# Load q_x from checkpoint and apply the precomputed scale

dequant_x = static_dequant(q_x, static_scales)

Key Differences Between Dynamic and Static Schemes

Scale Computation Timing: Dynamic schemes compute s = amax / 448 during inference for every block, while static schemes load pre-computed values from disk.

Memory and Latency Overhead: Dynamic quantization requires passing per-block scale tensors between layers, adding small memory reads and computational overhead. Static quantization minimizes runtime work by reusing stored scales but sacrifices adaptability.

Accuracy Characteristics: Dynamic quantization adapts to the current input distribution, often yielding higher accuracy when activation statistics vary across different generation steps. Static quantization performs best when inference data matches the calibration distribution exactly.

Implementation Status: DeepSeek-V3 exclusively implements dynamic quantization. The quantization_config in the released FP8 weight files explicitly declares "activation_scheme": "dynamic" as documented in README_WEIGHTS.md (lines 66-73). Static scheme code paths that bypass act_quant are not present in the current source.

Summary

• Dynamic quantization computes per-block scales at runtime via act_quant in inference/kernel.py, offering superior adaptability to input distribution shifts.
• Static quantization applies pre-calibrated scales stored with model weights, minimizing computational overhead but requiring distribution stability.
• DeepSeek-V3 adopts the dynamic scheme exclusively for its FP8 inference, using 128-channel blocks and the torch.float8_e4m3fn format.
• The configuration schema supports "activation_scheme": "static", though the corresponding implementation path is not included in the current release.

Frequently Asked Questions

What is the primary advantage of dynamic activation quantization?

Dynamic quantization adapts to the current input distribution by computing scales on-the-fly, often yielding higher accuracy when activation statistics vary across different inputs or generation steps. This flexibility comes at the cost of small per-inference computational overhead for calculating amax values.

Why does DeepSeek-V3 use FP8 E4M3FN for quantized activations?

The E4M3FN format provides 4 exponent bits and 3 mantissa bits with finite normalization, offering a better trade-off between range and precision for neural network activations compared to other 8-bit floating-point variants. The act_quant function in inference/kernel.py returns tensors with this specific dtype.

How does the block size parameter affect quantization accuracy?

The default block size of 128 channels determines the granularity of scale computation, where each block receives its own scale factor derived from amax / 448. Smaller blocks increase granularity and potentially improve accuracy but require more memory for storing per-block scales, while larger blocks reduce overhead at the cost of quantization fidelity.

Can I use static activation quantization with DeepSeek-V3 weights?

The current release does not provide static quantization implementations. While the configuration schema in README_WEIGHTS.md allows specifying "activation_scheme": "static", the source code lacks the necessary code paths to load and apply pre-computed activation scales, making dynamic quantization the only functional option.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}