# How NVFP4 Quantization Works in LongLive 2.0 for Transformer Inference

> Explore NVFP4 quantization in LongLive 2.0 for transformer inference. Discover how this 4-bit floating-point technique offers 8x memory savings with minimal quality loss.

- Repository: [NVIDIA Research Projects/LongLive](https://github.com/NVlabs/LongLive)
- Tags: deep-dive
- Published: 2026-05-24

---

**NVFP4 quantization in LongLive 2.0 compresses transformer weights to 4-bit floating-point (e4m3) using the FourOverSix library and custom Triton kernels, achieving approximately 8× memory reduction while maintaining FP16/BF16 quality through per-block scaling and runtime de-quantization.**

The **NVlabs/LongLive** repository implements **NVFP4 quantization** to enable ultra-efficient transformer inference. This 4-bit floating-point format, officially known as `e4m3`, powers the LongLive 2.0 generator by reducing memory footprint through the **FourOverSix (Fo6)** quantization library and a custom Triton kernel stack. The system stores weights in packed `uint8` containers and reconstructs them on-the-fly during attention operations.

## Understanding the NVFP4 Format and Architecture

**NVFP4** (also referred to as `e4m3`) is NVIDIA's 4-bit floating-point specification that stores values with 4 exponent bits and 3 mantissa bits. In LongLive 2.0, this format is implemented through the **FourOverSix (Fo6)** library, which packs two FP4 values into a single `uint8` element. The system computes **per-block scaling factors** using the rule `scale = amax / 6.0`, where `amax` represents the maximum absolute value within each quantization block.

When **Transformer Engine (TE)** integration is enabled, the pipeline optionally quantizes these per-block scales to FP8 using `scaled_e4m3_impl`, creating a hierarchical quantization scheme that maintains precision while minimizing storage overhead.

## The Three-Stage Quantization Pipeline

### Stage 1: Weight Quantization (FP32 to NVFP4)

The quantization process begins in [`fouroversix/src/fouroversix/quantize/triton/kernel.py`](https://github.com/NVlabs/LongLive/blob/main/fouroversix/src/fouroversix/quantize/triton/kernel.py), specifically within the `quantize_to_fp4` function (lines 45-73). This kernel performs four critical operations:

1. Computes the **maximum absolute value** (`amax`) for each block
2. Derives **per-block scales** using the static-6 scaling rule
3. Packs two FP4 values into each `uint8` element
4. Generates a **global `amax` scalar** for the entire tensor

For TE compatibility, the `static_blockwise_fp4_fake_quant` function handles the FP8 quantization of block scales, ensuring the representation fits into 4-bit weights plus 8-bit scaling factors.

### Stage 2: Model Integration and Checkpointing

Model-level quantization is orchestrated by `quantize_model_for_fouroversix_nvfp4` in [`utils/nvfp4_checkpoint.py`](https://github.com/NVlabs/LongLive/blob/main/utils/nvfp4_checkpoint.py) (line 86). This function traverses the model graph and replaces `nn.Linear` layers with quantization-aware wrappers. When `use_transformer_engine=True`, layers are wrapped with `TransformerEngineLinear` from [`utils/quant.py`](https://github.com/NVlabs/LongLive/blob/main/utils/quant.py), which forwards operations through TE's `autocast`/`fp8_autocast` context.

The checkpoint format is identified by the constant `NVFP4_CHECKPOINT_FORMAT = "longlive_generator_nvfp4"`, storing packed tensors alongside their per-block scales and global amax values.

### Stage 3: KV-Cache De-quantization at Inference

During text generation, the KV-cache stores tensors as `QuantizedTensor` objects containing packed `uint8` data and scaling factors. The `dequantize_kv_cache` function in [`utils/quant.py`](https://github.com/NVlabs/LongLive/blob/main/utils/quant.py) (lines 31-68) reconstructs dense tensors through three steps:

1. Building a 2D scale view from flattened per-block scales using `from_blocked`
2. Launching the Triton kernel `fp4_dequantize_kernel` from [`utils/nvfp4_kernel.py`](https://github.com/NVlabs/LongLive/blob/main/utils/nvfp4_kernel.py) (lines 41-67)
3. Applying the global scale calculation: `global_scale = amax / (e2m1_max * e4m3_max)`

The kernel expands each packed byte into two FP4 values, multiplies by per-block and global scales, and outputs `torch.float32` or `bf16` tensors reshaped to `[block_token_size, num_heads, 128]`.

## Code Implementation: Quantizing and Running Inference

### Quantizing with the FourOverSix Path

Use `quantize_model_for_fouroversix_nvfp4` when you need direct control over the Fo6 quantization pipeline:

```python
from utils.nvfp4_checkpoint import quantize_model_for_fouroversix_nvfp4
from utils.config import ModelQuantizationConfig

cfg = ModelQuantizationConfig(
    dtype="nvfp4",
    scale_rule="static_6",
)

quantized_model, matched = quantize_model_for_fouroversix_nvfp4(
    generator.model, cfg, keep_master_weights=False, verbose=True
)

```

### Integrating with Transformer Engine

For automatic precision management through NVIDIA's Transformer Engine:

```python
from utils.quant import quantize_model_with_filter

model, _ = quantize_model_with_filter(
    generator.model,
    quant_config={"dtype": "nvfp4"},
    use_transformer_engine=True,
    te_inference_only=True,
    te_low_precision_weights=True,
)

```

### Runtime KV-Cache De-quantization

During the generation loop, de-quantize KV-cache blocks before attention computation:

```python
from utils.quant import dequantize_kv_cache

dense_kv = dequantize_kv_cache(
    kv_list,
    max_blocks=128,
    num_heads=32,
    block_token_size=16,
    dtype=torch.bfloat16,
    device="cuda",
)

```

## Key Source Files and Kernel Architecture

- **[`fouroversix/src/fouroversix/quantize/triton/kernel.py`](https://github.com/NVlabs/LongLive/blob/main/fouroversix/src/fouroversix/quantize/triton/kernel.py)**: Core FP4 quantization logic including `quantize_to_fp4` and `static_blockwise_fp4_fake_quant`
- **[`utils/nvfp4_kernel.py`](https://github.com/NVlabs/LongLive/blob/main/utils/nvfp4_kernel.py)**: Triton de-quantization kernels, specifically `fp4_dequantize_kernel` for expanding packed bytes
- **[`utils/nvfp4_checkpoint.py`](https://github.com/NVlabs/LongLive/blob/main/utils/nvfp4_checkpoint.py)**: Checkpoint format definitions and the primary entry point `quantize_model_for_fouroversix_nvfp4`
- **[`utils/quant.py`](https://github.com/NVlabs/LongLive/blob/main/utils/quant.py)**: High-level quantization utilities, TE wrappers (`TransformerEngineLinear`), and KV-cache management via `dequantize_kv_cache`
- **[`utils/kernel/kv_dequant.py`](https://github.com/NVlabs/LongLive/blob/main/utils/kernel/kv_dequant.py)**: CUDA fallback kernels for KV-cache de-quantization when Triton is unavailable

## Summary

- **NVFP4** (`e4m3`) provides 4-bit floating-point storage with 4 exponent and 3 mantissa bits, implemented in LongLive 2.0 through the FourOverSix library
- The **packing mechanism** stores two FP4 values per `uint8` with per-block scaling computed as `amax / 6.0`
- **Triton kernels** handle both the forward quantization (`quantize_to_fp4`) and runtime de-quantization (`fp4_dequantize_kernel`)
- The system supports both **native Fo6 paths** and **Transformer Engine** wrappers for automatic precision casting
- Implementation achieves **approximately 8× memory reduction** compared to FP16/BF16 baselines while maintaining equivalent generation quality

## Frequently Asked Questions

### What is the difference between NVFP4 and standard INT4 quantization?

NVFP4 uses a 4-bit floating-point format (`e4m3`) with dynamic per-block scaling factors, whereas INT4 uses fixed-point integer representation. The floating-point format preserves better dynamic range for transformer weights through the scaling rule `scale = amax / 6.0`, as implemented in [`fouroversix/src/fouroversix/quantize/triton/kernel.py`](https://github.com/NVlabs/LongLive/blob/main/fouroversix/src/fouroversix/quantize/triton/kernel.py). This allows NVFP4 to handle weight distributions with outliers more effectively than uniform quantization.

### How does LongLive 2.0 handle the KV-cache with NVFP4 quantization?

The KV-cache stores quantized blocks as `QuantizedTensor` objects containing packed `uint8` data and per-block scales. During inference, `dequantize_kv_cache` in [`utils/quant.py`](https://github.com/NVlabs/LongLive/blob/main/utils/quant.py) reconstructs dense tensors by launching the Triton kernel `fp4_dequantize_kernel` from [`utils/nvfp4_kernel.py`](https://github.com/NVlabs/LongLive/blob/main/utils/nvfp4_kernel.py). The reconstruction applies both per-block scales and a global scaling factor calculated as `amax / (e2m1_max * e4m3_max)`.

### Can I use NVFP4 quantization without Transformer Engine?

Yes. Set `use_transformer_engine=False` when calling `quantize_model_for_fouroversix_nvfp4` in [`utils/nvfp4_checkpoint.py`](https://github.com/NVlabs/LongLive/blob/main/utils/nvfp4_checkpoint.py). This path uses the native FourOverSix implementation with custom Triton kernels rather than TE's `autocast` contexts, giving you explicit control over the quantization and de-quantization stages.

### What is the memory savings of NVFP4 compared to BF16?

NVFP4 achieves approximately **8× memory reduction** compared to BF16 or FP16 formats. This calculation accounts for the packed 4-bit weights plus the minimal overhead from per-block scaling tensors and the global amax scalar stored for each layer.