How to Use KV Cache Quantization with LongLive's LongLiveQuantizationConfig

Enable KV cache quantization in NVlabs/LongLive by setting --kv_quant true when running inference, or programmatically instantiate LongLiveQuantizationConfig with type="kv" to compress key-value tensors to FP4 precision using the FourOverSix pipeline.

LongLive is NVIDIA's open-source framework for efficient long-duration video generation using causal diffusion transformers. By configuring LongLiveQuantizationConfig specifically for KV cache quantization, you can significantly reduce GPU memory bandwidth and storage requirements during inference. The system automatically handles FP4 compression in the attention mechanism and de-quantizes tensors back to bfloat16 using optimized CUDA kernels.

Configuring LongLiveQuantizationConfig for KV Cache Quantization

The LongLiveQuantizationConfig dataclass in utils/quant.py (lines 75-88) serves as the primary interface for enabling KV cache quantization. This configuration subclasses the generic QuantizationConfig used by the FourOverSix pipeline and extends it with parameters specific to key-value memory compression.

When you initialize the config with type="kv", the pipeline automatically identifies incoming tensors as KV cache blocks rather than weights or activations. This triggers the FP4 quantization pathway that hooks into the causal diffusion model's attention mechanism.

Implementation Methods

You can enable KV cache quantization through command-line arguments or direct Python instantiation.

Command-Line Interface

The simplest approach relies on flags processed by CausalDiffusionInferencePipeline in pipeline/causal_diffusion_inference.py (lines 88-98). Pass the following arguments when running inference:

python -m main.inference \
  --model_kwargs.model_name Wan2.2-TI2V-5B \
  --kv_quant true \
  --kv_quant_scale_rule mse \
  --kv_quant_backend cuda \
  --inference.sampling_steps 50

The pipeline automatically constructs a LongLiveQuantizationConfig instance with type="kv" and attaches it to the underlying diffusion model.

Programmatic Configuration

For custom training or inference scripts, import the configuration class directly from the utilities module:

from utils.quant import LongLiveQuantizationConfig
from pipeline.causal_diffusion_inference import CausalDiffusionInferencePipeline

# Configure KV cache quantization manually

kv_cfg = LongLiveQuantizationConfig(
    scale_rule="static_6",  # Alternative: "mse", "static_4"

    backend="cuda",
    type="kv",
)

# Initialize pipeline and attach config

pipeline = CausalDiffusionInferencePipeline(args, device="cuda")
pipeline._dit_model.kv_quant_config = kv_cfg

# Run inference with compressed KV caches

video = pipeline.inference(noise, ["a cat walking"], return_latents=False)

How KV Cache Quantization Works Under the Hood

Understanding the quantization pipeline helps diagnose memory usage and potential precision issues.

FP4 Quantization in the Causal Model

During the forward pass, the causal transformer in wan_5b/modules/causal_model.py processes KV cache blocks through the quantize_to_fp4 function (line 1208). Each key and value block gets compressed independently:

# Simplified representation of the quantization loop

# Located in wan_5b/modules/causal_model.py lines 1208-1214

quantized_k = quantize_to_fp4(k_block, self.kv_quant_config)
quantized_v = quantize_to_fp4(v_block, self.kv_quant_config)

The QuantizeToFP4 operation applies per-block scaling factors determined by the scale_rule parameter, reducing memory footprint by approximately 4x compared to bfloat16 storage.

CUDA De-quantization Pipeline

Before attention computation, the compressed tensors must return to full precision. The dequantize_kv_cache_fp4 function in utils/kernel/kv_dequant.py (lines 53-82) dispatches a fused CUDA kernel that reconstructs bfloat16 values on-the-fly:

# From utils/kernel/kv_dequant.py

dequantize_kv_cache_fp4(quantized_cache, scale_factors, output_dtype=torch.bfloat16)

This de-quantization happens lazily during the attention operation, minimizing memory bandwidth while preserving computational accuracy.

Configuration Parameters Reference

The quantization behavior is controlled through specific fields in LongLiveQuantizationConfig:

• scale_rule: Determines how per-block scaling factors are calculated. Common options include "mse" (mean squared error optimized) and "static_6" (fixed denominator limits). The rule translates to FP4 denominator limits via scale_rule_to_fp4_limits in kv_dequant.py.
• backend: Currently only "cuda" is supported because the de-quantization kernel requires CUDA-specific optimizations.
• type: Must be explicitly set to "kv" to trigger KV cache quantization pathways rather than weight quantization.
• dtype: Inherits from the base config but defaults to torch.bfloat16 for the de-quantized representation used during attention computation.

Limitations and Compatibility Constraints

KV cache quantization is explicitly disabled when using Sequence-Parallel (Ulysses-SP) inference. The codebase in pipeline/causal_diffusion_inference_sp.py (line 200) raises an error if quantization is requested in SP mode, as the current implementation does not support the required communication patterns for compressed KV caches across parallel ranks.

To verify your KV cache is quantized during inference, inspect the cache objects after running a few diffusion steps:

# Access quantized KV cache from pipeline instance

k_block = pipeline.kv_cache_pos["k"][0]
print(f"Storage dtype: {k_block.values.dtype}")  # Shows FP4 packed format

print(f"Scale factors: {k_block.scale_factors.shape}")

Summary

• LongLiveQuantizationConfig in utils/quant.py controls FP4 KV cache compression via the type="kv" parameter.
• Enable quantization through CLI flags (--kv_quant true) or programmatic configuration before instantiating CausalDiffusionInferencePipeline.
• The causal model (wan_5b/modules/causal_model.py) applies quantize_to_fp4 to each KV block during the forward pass.
• A fused CUDA kernel in utils/kernel/kv_dequant.py handles on-the-fly de-quantization back to bf16 during attention computation.
• KV cache quantization is incompatible with Sequence-Parallel inference paths and will raise an error in pipeline/causal_diffusion_inference_sp.py.

Frequently Asked Questions

What memory savings does KV cache quantization provide?

KV cache quantization reduces memory bandwidth and storage by approximately 4x by compressing tensors from bfloat16 to FP4 format. Since KV caches dominate memory consumption in long-sequence video generation, this allows processing significantly longer videos or larger batch sizes while maintaining the same GPU memory footprint.

Can I use KV cache quantization with Sequence-Parallel inference?

No. According to the source code in pipeline/causal_diffusion_inference_sp.py (line 200), KV cache quantization is explicitly unsupported in the Sequence-Parallel (Ulysses-SP) inference path. The implementation does not handle the all-to-all communication patterns required for quantized tensors across parallel ranks, and attempting to enable it will raise a runtime error.

Which scale rule should I choose for my use case?

Choose "mse" when you prioritize reconstruction accuracy and can tolerate slightly higher computational overhead for dynamic scale factor calculation. Use "static_6" or similar static rules when you need deterministic memory access patterns and maximum throughput, as these use pre-defined denominator limits without per-block optimization loops.

Is the FP4 quantization lossless?

No, FP4 quantization is a lossy compression technique that reduces precision from 16-bit floating point to 4-bit representations. However, the FourOverSix pipeline used by LongLive applies sophisticated scaling rules (configured via scale_rule) that minimize perceptual distortion in the generated video output. The de-quantization kernel reconstructs bfloat16 values before attention computation, preventing accumulation of quantization errors across multiple diffusion steps.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}