How Block-wise FP8 Weight Quantization (128×128) Works in DeepSeek-V3

DeepSeek-V3 compresses model weights to FP8 by dividing matrices into 128×128 tiles, computing a per-block scaling factor for each tile, and de-quantizing on-the-fly during matrix multiplication to avoid materializing full-precision copies.

DeepSeek-V3 achieves high-throughput inference through an aggressive block-wise FP8 weight quantization scheme that preserves accuracy while minimizing memory bandwidth. This implementation, found in the inference/kernel.py module, processes weight matrices in 128×128 blocks (denoted as "I28×I28" in the source documentation) and utilizes Triton kernels to handle both quantization and quantized GEMM operations.

The 128×128 Block-wise Quantization Scheme

The quantization strategy treats a weight matrix W of shape (M, N) as a grid of independent 128×128 tiles. Each tile is quantized separately to the torch.float8_e4m3fn format, and a distinct scaling factor is computed and stored for every tile. This block-wise approach prevents outliers in one region of the matrix from degrading the precision of distant blocks, a significant advantage over per-channel or per-tensor quantization.

The "I28×I28" notation refers to the 128-element block dimensions (where "I" represents 1 and "28" represents 28, combining to form 128). Each dimension of the weight matrix must be divisible by 128 to align with this tiling scheme.

Kernel Implementation in inference/kernel.py

The quantization and de-quantization pipeline is implemented across three core components in inference/kernel.py: the low-level quantization kernel, the high-level API wrapper, and the fused GEMM kernel.

Stage 1: Per-Tile Scale Computation and Quantization (act_quant_kernel)

The act_quant_kernel Triton kernel (lines 10‑35) processes the input tensor as a contiguous 1‑D array divided into BLOCK_SIZE chunks of 128 elements.

For each block, the kernel executes the following steps:

1. Find the maximum absolute value (amax) across the 128 elements.
2. Clamp amax to a minimum of 1e-4 to avoid division by zero.
3. Compute the scale s = amax / 448.0, where 448 is the maximum representable value in the FP8 E4M3 format.
4. Optionally round the scale to a power-of-2 if scale_fmt="ue8m0" is specified, optimizing hardware operations.
5. Quantize each element by dividing by s and casting to torch.float8_e4m3fn.
6. Store the scale s to a separate scale tensor indexed by the block ID.

# Conceptual flow of act_quant_kernel operations

# Input: float32 tensor divided into 128-element blocks

# Output: FP8 tensor + per-block scale tensor

# Pseudocode representing the kernel logic:

for block_id in range(num_blocks):
    block = input[block_id * 128 : (block_id + 1) * 128]
    amax = max(abs(block))
    amax = max(amax, 1e-4)
    scale = amax / 448.0
    # Optional: round scale to power-of-2 (ue8m0)

    quantized_block = (block / scale).to(torch.float8_e4m3fn)
    output[block_id * 128 : (block_id + 1) * 128] = quantized_block
    scales[block_id] = scale

Stage 2: High-Level Weight Quantization API (act_quant)

The act_quant function (lines 38‑57) provides the Python interface for the quantization kernel. It validates that the last dimension of the input tensor is divisible by the block_size (default 128), allocates the output FP8 tensor and the scale tensor of shape (..., N // block_size), and launches the Triton kernel with a grid covering the full element count.

from inference.kernel import act_quant
import torch

# Example: Quantizing a linear layer weight matrix

weight = torch.randn(4096, 4096, dtype=torch.float32, device='cuda')

# Quantize with 128x128 block-wise FP8

q_weight, weight_scale = act_quant(weight, block_size=128)

print(f"Quantized dtype: {q_weight.dtype}")  # torch.float8_e4m3fn

print(f"Scale shape: {weight_scale.shape}")   # (4096//128, 4096//128) = (32, 32)

Stage 3: FP8 GEMM with On-the-Fly De-quantization (fp8_gemm_kernel)

The fp8_gemm_kernel (lines 120‑172) performs matrix multiplication on the quantized tensors without materializing full-precision copies. It receives the quantized matrices a and b along with their per-tile scale tensors a_s and b_s.

The kernel processes the K-dimension in tiles of BLOCK_SIZE_K = 128. For each tile:

1. Load FP8 values from the current K-tile of matrices a and b.
2. De-quantize on-the-fly by multiplying with the corresponding scales: a_val * a_s[:, None] * b_s[None, :].
3. Accumulate the FP32 dot product across the K-tile.
4. Cast the final accumulator to the output dtype.

This approach minimizes memory bandwidth by keeping weights in FP8 format throughout the computation, only expanding to FP32 within the GPU registers during the accumulation phase.

from inference.kernel import fp8_gemm, act_quant
import torch

# Quantized weight (from previous example)

weight = torch.randn(4096, 4096, device='cuda', dtype=torch.float32)
q_weight, w_scale = act_quant(weight, block_size=128)

# Quantized activation

activation = torch.randn(1, 4096, device='cuda', dtype=torch.float32)
q_act, a_scale = act_quant(activation, block_size=128)

# FP8 matrix multiplication with automatic de-quantization

# Note: weight must be transposed for the GEMM

output = fp8_gemm(q_act, a_scale, q_weight.t(), w_scale.t())
print(output.shape)  # torch.Size([1, 4096])

Supporting Files and Data Format

The quantization scheme relies on specific metadata stored alongside the model weights. According to the repository's README_WEIGHTS.md, the FP8 weights are accompanied by a quantization_config and weight_scale_inv fields that store the per-tile scales. The inference/fp8_cast_bf16.py utility demonstrates how these scales are applied to convert stored FP8 weights back to BF16 for validation or comparison purposes.

Summary

• Block-wise FP8 weight quantization in DeepSeek-V3 processes weight matrices in 128×128 tiles to isolate outliers and preserve precision.
• The act_quant_kernel computes per-block scales as amax / 448 and quantizes to torch.float8_e4m3fn, storing scales separately.
• The act_quant Python API validates dimensions and orchestrates the Triton kernel launch for both weights and activations.
• The fp8_gemm_kernel performs matrix multiplication without materializing full-precision weights, de-quantizing tiles on-the-fly using the stored per-block scales.
• This implementation in inference/kernel.py enables efficient inference by minimizing memory bandwidth while maintaining model accuracy through fine-grained block-wise scaling.

Frequently Asked Questions

What does the 128×128 block size refer to in DeepSeek-V3's quantization?

The 128×128 block size, denoted as "I28×I28" in the source documentation, refers to square tiles of 128 elements in each dimension. A weight matrix is divided into these tiles, and each tile is quantized independently with its own scaling factor. This granularity prevents outliers in one tile from reducing the precision of other tiles, which would occur with per-tensor quantization.

Why is the scale computed as amax / 448 in the quantization kernel?

The value 448 represents the maximum representable value in the FP8 E4M3 format (torch.float8_e4m3fn). By dividing the maximum absolute value (amax) of a 128-element block by 448, the kernel determines the scaling factor needed to map the block's dynamic range onto the FP8 representable range. The kernel also clamps amax to a minimum of 1e-4 to prevent division by zero.

How does the FP8 GEMM kernel avoid the memory overhead of de-quantizing weights?

The fp8_gemm_kernel processes the matrix multiplication in tiles of 128 along the K-dimension. For each tile, it loads the FP8 values into registers, multiplies them by their corresponding per-tile scales (a_s and b_s) to convert to FP32 on-the-fly, and immediately accumulates the dot product. This fused approach keeps weights in FP8 format in global memory throughout the computation, only expanding to full precision temporarily within the GPU's compute units.

What is the purpose of the optional ue8m0 scale format?

The ue8m0 format is an optional configuration that rounds the computed scale to a power-of-two value. This optimization simplifies hardware operations by converting the scale multiplication into a bit-shift operation, potentially improving performance on specific hardware targets. When enabled, the kernel adjusts the scale calculation to ensure the resulting value is a power of two before applying it to the quantization formula.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}