How NVFP4 Quantization Works in LongLive 2.0 for Transformer Inference

NVFP4 quantization in LongLive 2.0 compresses transformer weights to 4-bit floating-point (e4m3) using the FourOverSix library and custom Triton kernels, achieving approximately 8× memory reduction while maintaining FP16/BF16 quality through per-block scaling and runtime de-quantization.

The NVlabs/LongLive repository implements NVFP4 quantization to enable ultra-efficient transformer inference. This 4-bit floating-point format, officially known as e4m3, powers the LongLive 2.0 generator by reducing memory footprint through the FourOverSix (Fo6) quantization library and a custom Triton kernel stack. The system stores weights in packed uint8 containers and reconstructs them on-the-fly during attention operations.

Understanding the NVFP4 Format and Architecture

NVFP4 (also referred to as e4m3) is NVIDIA's 4-bit floating-point specification that stores values with 4 exponent bits and 3 mantissa bits. In LongLive 2.0, this format is implemented through the FourOverSix (Fo6) library, which packs two FP4 values into a single uint8 element. The system computes per-block scaling factors using the rule scale = amax / 6.0, where amax represents the maximum absolute value within each quantization block.

When Transformer Engine (TE) integration is enabled, the pipeline optionally quantizes these per-block scales to FP8 using scaled_e4m3_impl, creating a hierarchical quantization scheme that maintains precision while minimizing storage overhead.

The Three-Stage Quantization Pipeline

Stage 1: Weight Quantization (FP32 to NVFP4)

The quantization process begins in fouroversix/src/fouroversix/quantize/triton/kernel.py, specifically within the quantize_to_fp4 function (lines 45-73). This kernel performs four critical operations:

1. Computes the maximum absolute value (amax) for each block
2. Derives per-block scales using the static-6 scaling rule
3. Packs two FP4 values into each uint8 element
4. Generates a global amax scalar for the entire tensor

For TE compatibility, the static_blockwise_fp4_fake_quant function handles the FP8 quantization of block scales, ensuring the representation fits into 4-bit weights plus 8-bit scaling factors.

Stage 2: Model Integration and Checkpointing

Model-level quantization is orchestrated by quantize_model_for_fouroversix_nvfp4 in utils/nvfp4_checkpoint.py (line 86). This function traverses the model graph and replaces nn.Linear layers with quantization-aware wrappers. When use_transformer_engine=True, layers are wrapped with TransformerEngineLinear from utils/quant.py, which forwards operations through TE's autocast/fp8_autocast context.

The checkpoint format is identified by the constant NVFP4_CHECKPOINT_FORMAT = "longlive_generator_nvfp4", storing packed tensors alongside their per-block scales and global amax values.

Stage 3: KV-Cache De-quantization at Inference

During text generation, the KV-cache stores tensors as QuantizedTensor objects containing packed uint8 data and scaling factors. The dequantize_kv_cache function in utils/quant.py (lines 31-68) reconstructs dense tensors through three steps:

1. Building a 2D scale view from flattened per-block scales using from_blocked
2. Launching the Triton kernel fp4_dequantize_kernel from utils/nvfp4_kernel.py (lines 41-67)
3. Applying the global scale calculation: global_scale = amax / (e2m1_max * e4m3_max)

The kernel expands each packed byte into two FP4 values, multiplies by per-block and global scales, and outputs torch.float32 or bf16 tensors reshaped to [block_token_size, num_heads, 128].

Code Implementation: Quantizing and Running Inference

Quantizing with the FourOverSix Path

Use quantize_model_for_fouroversix_nvfp4 when you need direct control over the Fo6 quantization pipeline:

from utils.nvfp4_checkpoint import quantize_model_for_fouroversix_nvfp4
from utils.config import ModelQuantizationConfig

cfg = ModelQuantizationConfig(
    dtype="nvfp4",
    scale_rule="static_6",
)

quantized_model, matched = quantize_model_for_fouroversix_nvfp4(
    generator.model, cfg, keep_master_weights=False, verbose=True
)

Integrating with Transformer Engine

For automatic precision management through NVIDIA's Transformer Engine:

from utils.quant import quantize_model_with_filter

model, _ = quantize_model_with_filter(
    generator.model,
    quant_config={"dtype": "nvfp4"},
    use_transformer_engine=True,
    te_inference_only=True,
    te_low_precision_weights=True,
)

Runtime KV-Cache De-quantization

During the generation loop, de-quantize KV-cache blocks before attention computation:

from utils.quant import dequantize_kv_cache

dense_kv = dequantize_kv_cache(
    kv_list,
    max_blocks=128,
    num_heads=32,
    block_token_size=16,
    dtype=torch.bfloat16,
    device="cuda",
)

Key Source Files and Kernel Architecture

• fouroversix/src/fouroversix/quantize/triton/kernel.py: Core FP4 quantization logic including quantize_to_fp4 and static_blockwise_fp4_fake_quant
• utils/nvfp4_kernel.py: Triton de-quantization kernels, specifically fp4_dequantize_kernel for expanding packed bytes
• utils/nvfp4_checkpoint.py: Checkpoint format definitions and the primary entry point quantize_model_for_fouroversix_nvfp4
• utils/quant.py: High-level quantization utilities, TE wrappers (TransformerEngineLinear), and KV-cache management via dequantize_kv_cache
• utils/kernel/kv_dequant.py: CUDA fallback kernels for KV-cache de-quantization when Triton is unavailable

Summary

• NVFP4 (e4m3) provides 4-bit floating-point storage with 4 exponent and 3 mantissa bits, implemented in LongLive 2.0 through the FourOverSix library
• The packing mechanism stores two FP4 values per uint8 with per-block scaling computed as amax / 6.0
• Triton kernels handle both the forward quantization (quantize_to_fp4) and runtime de-quantization (fp4_dequantize_kernel)
• The system supports both native Fo6 paths and Transformer Engine wrappers for automatic precision casting
• Implementation achieves approximately 8× memory reduction compared to FP16/BF16 baselines while maintaining equivalent generation quality

Frequently Asked Questions

What is the difference between NVFP4 and standard INT4 quantization?

NVFP4 uses a 4-bit floating-point format (e4m3) with dynamic per-block scaling factors, whereas INT4 uses fixed-point integer representation. The floating-point format preserves better dynamic range for transformer weights through the scaling rule scale = amax / 6.0, as implemented in fouroversix/src/fouroversix/quantize/triton/kernel.py. This allows NVFP4 to handle weight distributions with outliers more effectively than uniform quantization.

How does LongLive 2.0 handle the KV-cache with NVFP4 quantization?

The KV-cache stores quantized blocks as QuantizedTensor objects containing packed uint8 data and per-block scales. During inference, dequantize_kv_cache in utils/quant.py reconstructs dense tensors by launching the Triton kernel fp4_dequantize_kernel from utils/nvfp4_kernel.py. The reconstruction applies both per-block scales and a global scaling factor calculated as amax / (e2m1_max * e4m3_max).

Can I use NVFP4 quantization without Transformer Engine?

Yes. Set use_transformer_engine=False when calling quantize_model_for_fouroversix_nvfp4 in utils/nvfp4_checkpoint.py. This path uses the native FourOverSix implementation with custom Triton kernels rather than TE's autocast contexts, giving you explicit control over the quantization and de-quantization stages.

What is the memory savings of NVFP4 compared to BF16?

NVFP4 achieves approximately 8× memory reduction compared to BF16 or FP16 formats. This calculation accounts for the packed 4-bit weights plus the minimal overhead from per-block scaling tensors and the global amax scalar stored for each layer.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}