# How Block-wise FP8 Weight Quantization (128×128) Works in DeepSeek-V3

> Discover how DeepSeek-V3 uses block-wise FP8 weight quantization (128x128) in its kernel. Learn about per-block scaling and on-the-fly de-quantization for efficient matrix multiplication.

- Repository: [DeepSeek/DeepSeek-V3](https://github.com/deepseek-ai/DeepSeek-V3)
- Tags: internals
- Published: 2026-02-26

---

**DeepSeek-V3 compresses model weights to FP8 by dividing matrices into 128×128 tiles, computing a per-block scaling factor for each tile, and de-quantizing on-the-fly during matrix multiplication to avoid materializing full-precision copies.**

DeepSeek-V3 achieves high-throughput inference through an aggressive **block-wise FP8 weight quantization** scheme that preserves accuracy while minimizing memory bandwidth. This implementation, found in the [`inference/kernel.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/kernel.py) module, processes weight matrices in 128×128 blocks (denoted as "I28×I28" in the source documentation) and utilizes Triton kernels to handle both quantization and quantized GEMM operations.

## The 128×128 Block-wise Quantization Scheme

The quantization strategy treats a weight matrix `W` of shape `(M, N)` as a grid of independent **128×128 tiles**. Each tile is quantized separately to the `torch.float8_e4m3fn` format, and a distinct scaling factor is computed and stored for every tile. This block-wise approach prevents outliers in one region of the matrix from degrading the precision of distant blocks, a significant advantage over per-channel or per-tensor quantization.

The "I28×I28" notation refers to the 128-element block dimensions (where "I" represents 1 and "28" represents 28, combining to form 128). Each dimension of the weight matrix must be divisible by 128 to align with this tiling scheme.

## Kernel Implementation in [`inference/kernel.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/kernel.py)

The quantization and de-quantization pipeline is implemented across three core components in [`inference/kernel.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/kernel.py): the low-level quantization kernel, the high-level API wrapper, and the fused GEMM kernel.

### Stage 1: Per-Tile Scale Computation and Quantization (`act_quant_kernel`)

The `act_quant_kernel` Triton kernel (lines [10‑35](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/kernel.py#L10-L35)) processes the input tensor as a contiguous 1‑D array divided into `BLOCK_SIZE` chunks of 128 elements.

For each block, the kernel executes the following steps:

1. **Find the maximum absolute value (`amax`)** across the 128 elements.
2. **Clamp `amax`** to a minimum of `1e-4` to avoid division by zero.
3. **Compute the scale** `s = amax / 448.0`, where 448 is the maximum representable value in the FP8 E4M3 format.
4. **Optionally round the scale** to a power-of-2 if `scale_fmt="ue8m0"` is specified, optimizing hardware operations.
5. **Quantize each element** by dividing by `s` and casting to `torch.float8_e4m3fn`.
6. **Store the scale** `s` to a separate scale tensor indexed by the block ID.

```python

# Conceptual flow of act_quant_kernel operations

# Input: float32 tensor divided into 128-element blocks

# Output: FP8 tensor + per-block scale tensor

# Pseudocode representing the kernel logic:

for block_id in range(num_blocks):
    block = input[block_id * 128 : (block_id + 1) * 128]
    amax = max(abs(block))
    amax = max(amax, 1e-4)
    scale = amax / 448.0
    # Optional: round scale to power-of-2 (ue8m0)

    quantized_block = (block / scale).to(torch.float8_e4m3fn)
    output[block_id * 128 : (block_id + 1) * 128] = quantized_block
    scales[block_id] = scale

```

### Stage 2: High-Level Weight Quantization API (`act_quant`)

The `act_quant` function (lines [38‑57](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/kernel.py#L38-L57)) provides the Python interface for the quantization kernel. It validates that the last dimension of the input tensor is divisible by the `block_size` (default 128), allocates the output FP8 tensor and the scale tensor of shape `(..., N // block_size)`, and launches the Triton kernel with a grid covering the full element count.

```python
from inference.kernel import act_quant
import torch

# Example: Quantizing a linear layer weight matrix

weight = torch.randn(4096, 4096, dtype=torch.float32, device='cuda')

# Quantize with 128x128 block-wise FP8

q_weight, weight_scale = act_quant(weight, block_size=128)

print(f"Quantized dtype: {q_weight.dtype}")  # torch.float8_e4m3fn

print(f"Scale shape: {weight_scale.shape}")   # (4096//128, 4096//128) = (32, 32)

```

### Stage 3: FP8 GEMM with On-the-Fly De-quantization (`fp8_gemm_kernel`)

The `fp8_gemm_kernel` (lines [120‑172](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/kernel.py#L120-L172)) performs matrix multiplication on the quantized tensors without materializing full-precision copies. It receives the quantized matrices `a` and `b` along with their per-tile scale tensors `a_s` and `b_s`.

The kernel processes the K-dimension in tiles of `BLOCK_SIZE_K = 128`. For each tile:

1. **Load FP8 values** from the current K-tile of matrices `a` and `b`.
2. **De-quantize on-the-fly** by multiplying with the corresponding scales: `a_val * a_s[:, None] * b_s[None, :]`.
3. **Accumulate** the FP32 dot product across the K-tile.
4. **Cast the final accumulator** to the output dtype.

This approach minimizes memory bandwidth by keeping weights in FP8 format throughout the computation, only expanding to FP32 within the GPU registers during the accumulation phase.

```python
from inference.kernel import fp8_gemm, act_quant
import torch

# Quantized weight (from previous example)

weight = torch.randn(4096, 4096, device='cuda', dtype=torch.float32)
q_weight, w_scale = act_quant(weight, block_size=128)

# Quantized activation

activation = torch.randn(1, 4096, device='cuda', dtype=torch.float32)
q_act, a_scale = act_quant(activation, block_size=128)

# FP8 matrix multiplication with automatic de-quantization

# Note: weight must be transposed for the GEMM

output = fp8_gemm(q_act, a_scale, q_weight.t(), w_scale.t())
print(output.shape)  # torch.Size([1, 4096])

```

## Supporting Files and Data Format

The quantization scheme relies on specific metadata stored alongside the model weights. According to the repository's [`README_WEIGHTS.md`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/README_WEIGHTS.md), the FP8 weights are accompanied by a `quantization_config` and `weight_scale_inv` fields that store the per-tile scales. The [`inference/fp8_cast_bf16.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/fp8_cast_bf16.py) utility demonstrates how these scales are applied to convert stored FP8 weights back to BF16 for validation or comparison purposes.

## Summary

- **Block-wise FP8 weight quantization** in DeepSeek-V3 processes weight matrices in 128×128 tiles to isolate outliers and preserve precision.
- The **`act_quant_kernel`** computes per-block scales as `amax / 448` and quantizes to `torch.float8_e4m3fn`, storing scales separately.
- The **`act_quant`** Python API validates dimensions and orchestrates the Triton kernel launch for both weights and activations.
- The **`fp8_gemm_kernel`** performs matrix multiplication without materializing full-precision weights, de-quantizing tiles on-the-fly using the stored per-block scales.
- This implementation in [`inference/kernel.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/kernel.py) enables efficient inference by minimizing memory bandwidth while maintaining model accuracy through fine-grained block-wise scaling.

## Frequently Asked Questions

### What does the 128×128 block size refer to in DeepSeek-V3's quantization?

The 128×128 block size, denoted as "I28×I28" in the source documentation, refers to square tiles of 128 elements in each dimension. A weight matrix is divided into these tiles, and each tile is quantized independently with its own scaling factor. This granularity prevents outliers in one tile from reducing the precision of other tiles, which would occur with per-tensor quantization.

### Why is the scale computed as `amax / 448` in the quantization kernel?

The value 448 represents the maximum representable value in the FP8 E4M3 format (`torch.float8_e4m3fn`). By dividing the maximum absolute value (`amax`) of a 128-element block by 448, the kernel determines the scaling factor needed to map the block's dynamic range onto the FP8 representable range. The kernel also clamps `amax` to a minimum of `1e-4` to prevent division by zero.

### How does the FP8 GEMM kernel avoid the memory overhead of de-quantizing weights?

The `fp8_gemm_kernel` processes the matrix multiplication in tiles of 128 along the K-dimension. For each tile, it loads the FP8 values into registers, multiplies them by their corresponding per-tile scales (`a_s` and `b_s`) to convert to FP32 on-the-fly, and immediately accumulates the dot product. This fused approach keeps weights in FP8 format in global memory throughout the computation, only expanding to full precision temporarily within the GPU's compute units.

### What is the purpose of the optional `ue8m0` scale format?

The `ue8m0` format is an optional configuration that rounds the computed scale to a power-of-two value. This optimization simplifies hardware operations by converting the scale multiplication into a bit-shift operation, potentially improving performance on specific hardware targets. When enabled, the kernel adjusts the scale calculation to ensure the resulting value is a power of two before applying it to the quantization formula.