# How to Use KV Cache Quantization with LongLive's LongLiveQuantizationConfig

> Learn to use KV cache quantization with LongLiveQuantizationConfig in NVlabs/LongLive. Compress key-value tensors to FP4 with ease for faster inference.

- Repository: [NVIDIA Research Projects/LongLive](https://github.com/NVlabs/LongLive)
- Tags: how-to-guide
- Published: 2026-05-24

---

**Enable KV cache quantization in NVlabs/LongLive by setting `--kv_quant true` when running inference, or programmatically instantiate `LongLiveQuantizationConfig` with `type="kv"` to compress key-value tensors to FP4 precision using the FourOverSix pipeline.**

LongLive is NVIDIA's open-source framework for efficient long-duration video generation using causal diffusion transformers. By configuring **`LongLiveQuantizationConfig`** specifically for KV cache quantization, you can significantly reduce GPU memory bandwidth and storage requirements during inference. The system automatically handles FP4 compression in the attention mechanism and de-quantizes tensors back to bfloat16 using optimized CUDA kernels.

## Configuring LongLiveQuantizationConfig for KV Cache Quantization

The **`LongLiveQuantizationConfig`** dataclass in [`utils/quant.py`](https://github.com/NVlabs/LongLive/blob/main/utils/quant.py) (lines 75-88) serves as the primary interface for enabling KV cache quantization. This configuration subclasses the generic `QuantizationConfig` used by the FourOverSix pipeline and extends it with parameters specific to key-value memory compression.

When you initialize the config with `type="kv"`, the pipeline automatically identifies incoming tensors as KV cache blocks rather than weights or activations. This triggers the FP4 quantization pathway that hooks into the causal diffusion model's attention mechanism.

## Implementation Methods

You can enable KV cache quantization through command-line arguments or direct Python instantiation.

### Command-Line Interface

The simplest approach relies on flags processed by `CausalDiffusionInferencePipeline` in [`pipeline/causal_diffusion_inference.py`](https://github.com/NVlabs/LongLive/blob/main/pipeline/causal_diffusion_inference.py) (lines 88-98). Pass the following arguments when running inference:

```bash
python -m main.inference \
  --model_kwargs.model_name Wan2.2-TI2V-5B \
  --kv_quant true \
  --kv_quant_scale_rule mse \
  --kv_quant_backend cuda \
  --inference.sampling_steps 50

```

The pipeline automatically constructs a `LongLiveQuantizationConfig` instance with `type="kv"` and attaches it to the underlying diffusion model.

### Programmatic Configuration

For custom training or inference scripts, import the configuration class directly from the utilities module:

```python
from utils.quant import LongLiveQuantizationConfig
from pipeline.causal_diffusion_inference import CausalDiffusionInferencePipeline

# Configure KV cache quantization manually

kv_cfg = LongLiveQuantizationConfig(
    scale_rule="static_6",  # Alternative: "mse", "static_4"

    backend="cuda",
    type="kv",
)

# Initialize pipeline and attach config

pipeline = CausalDiffusionInferencePipeline(args, device="cuda")
pipeline._dit_model.kv_quant_config = kv_cfg

# Run inference with compressed KV caches

video = pipeline.inference(noise, ["a cat walking"], return_latents=False)

```

## How KV Cache Quantization Works Under the Hood

Understanding the quantization pipeline helps diagnose memory usage and potential precision issues.

### FP4 Quantization in the Causal Model

During the forward pass, the causal transformer in [`wan_5b/modules/causal_model.py`](https://github.com/NVlabs/LongLive/blob/main/wan_5b/modules/causal_model.py) processes KV cache blocks through the `quantize_to_fp4` function (line 1208). Each key and value block gets compressed independently:

```python

# Simplified representation of the quantization loop

# Located in wan_5b/modules/causal_model.py lines 1208-1214

quantized_k = quantize_to_fp4(k_block, self.kv_quant_config)
quantized_v = quantize_to_fp4(v_block, self.kv_quant_config)

```

The `QuantizeToFP4` operation applies per-block scaling factors determined by the `scale_rule` parameter, reducing memory footprint by approximately 4x compared to bfloat16 storage.

### CUDA De-quantization Pipeline

Before attention computation, the compressed tensors must return to full precision. The `dequantize_kv_cache_fp4` function in [`utils/kernel/kv_dequant.py`](https://github.com/NVlabs/LongLive/blob/main/utils/kernel/kv_dequant.py) (lines 53-82) dispatches a fused CUDA kernel that reconstructs bfloat16 values on-the-fly:

```python

# From utils/kernel/kv_dequant.py

dequantize_kv_cache_fp4(quantized_cache, scale_factors, output_dtype=torch.bfloat16)

```

This de-quantization happens lazily during the attention operation, minimizing memory bandwidth while preserving computational accuracy.

## Configuration Parameters Reference

The quantization behavior is controlled through specific fields in `LongLiveQuantizationConfig`:

- **scale_rule**: Determines how per-block scaling factors are calculated. Common options include `"mse"` (mean squared error optimized) and `"static_6"` (fixed denominator limits). The rule translates to FP4 denominator limits via `scale_rule_to_fp4_limits` in [`kv_dequant.py`](https://github.com/NVlabs/LongLive/blob/main/kv_dequant.py).
- **backend**: Currently only `"cuda"` is supported because the de-quantization kernel requires CUDA-specific optimizations.
- **type**: Must be explicitly set to `"kv"` to trigger KV cache quantization pathways rather than weight quantization.
- **dtype**: Inherits from the base config but defaults to `torch.bfloat16` for the de-quantized representation used during attention computation.

## Limitations and Compatibility Constraints

**KV cache quantization is explicitly disabled** when using Sequence-Parallel (Ulysses-SP) inference. The codebase in [`pipeline/causal_diffusion_inference_sp.py`](https://github.com/NVlabs/LongLive/blob/main/pipeline/causal_diffusion_inference_sp.py) (line 200) raises an error if quantization is requested in SP mode, as the current implementation does not support the required communication patterns for compressed KV caches across parallel ranks.

To verify your KV cache is quantized during inference, inspect the cache objects after running a few diffusion steps:

```python

# Access quantized KV cache from pipeline instance

k_block = pipeline.kv_cache_pos["k"][0]
print(f"Storage dtype: {k_block.values.dtype}")  # Shows FP4 packed format

print(f"Scale factors: {k_block.scale_factors.shape}")

```

## Summary

- **`LongLiveQuantizationConfig`** in [`utils/quant.py`](https://github.com/NVlabs/LongLive/blob/main/utils/quant.py) controls FP4 KV cache compression via the `type="kv"` parameter.
- Enable quantization through CLI flags (`--kv_quant true`) or programmatic configuration before instantiating `CausalDiffusionInferencePipeline`.
- The causal model ([`wan_5b/modules/causal_model.py`](https://github.com/NVlabs/LongLive/blob/main/wan_5b/modules/causal_model.py)) applies `quantize_to_fp4` to each KV block during the forward pass.
- A fused CUDA kernel in [`utils/kernel/kv_dequant.py`](https://github.com/NVlabs/LongLive/blob/main/utils/kernel/kv_dequant.py) handles on-the-fly de-quantization back to `bf16` during attention computation.
- KV cache quantization is incompatible with Sequence-Parallel inference paths and will raise an error in [`pipeline/causal_diffusion_inference_sp.py`](https://github.com/NVlabs/LongLive/blob/main/pipeline/causal_diffusion_inference_sp.py).

## Frequently Asked Questions

### What memory savings does KV cache quantization provide?

KV cache quantization reduces memory bandwidth and storage by approximately 4x by compressing tensors from bfloat16 to FP4 format. Since KV caches dominate memory consumption in long-sequence video generation, this allows processing significantly longer videos or larger batch sizes while maintaining the same GPU memory footprint.

### Can I use KV cache quantization with Sequence-Parallel inference?

No. According to the source code in [`pipeline/causal_diffusion_inference_sp.py`](https://github.com/NVlabs/LongLive/blob/main/pipeline/causal_diffusion_inference_sp.py) (line 200), KV cache quantization is explicitly unsupported in the Sequence-Parallel (Ulysses-SP) inference path. The implementation does not handle the all-to-all communication patterns required for quantized tensors across parallel ranks, and attempting to enable it will raise a runtime error.

### Which scale rule should I choose for my use case?

Choose `"mse"` when you prioritize reconstruction accuracy and can tolerate slightly higher computational overhead for dynamic scale factor calculation. Use `"static_6"` or similar static rules when you need deterministic memory access patterns and maximum throughput, as these use pre-defined denominator limits without per-block optimization loops.

### Is the FP4 quantization lossless?

No, FP4 quantization is a lossy compression technique that reduces precision from 16-bit floating point to 4-bit representations. However, the FourOverSix pipeline used by LongLive applies sophisticated scaling rules (configured via `scale_rule`) that minimize perceptual distortion in the generated video output. The de-quantization kernel reconstructs bfloat16 values before attention computation, preventing accumulation of quantization errors across multiple diffusion steps.