Requirements for FP8 Scaled MM Quantization in LTX-2: A Complete Guide

LTX-2 requires pre-quantized checkpoints containing FP8 weights (F8_E4M3 dtype) paired with per-tensor scale tensors, plus a PyTorch build supporting torch.float8_e4m3fn, to utilize FP8 scaled-matrix-multiply (MM) quantization.

The LTX-2 video generation framework from Lightricks provides an optimized inference path using FP8 scaled MM quantization, but this capability depends on specific checkpoint characteristics and environment requirements. As implemented in the ltx_core.quantization module, this quantization method differs from runtime casting approaches by requiring weights that are already quantized to the F8_E4M3 format with pre-calculated scaling factors.

Prerequisites for FP8 Scaled MM Quantization

Pre-quantized Checkpoint Structure

The FP8 scaled MM path in LTX-2 strictly requires checkpoints that have been quantized ahead of time. According to the source code in packages/ltx-core/src/ltx_core/quantization/fp8_scaled_mm.py, your checkpoint must satisfy two structural conditions:

1. FP8 weight tensors – The safetensors file must contain weights with dtype F8_E4M3 (mapped to torch.float8_e4m3fn in PyTorch).
2. Per-tensor scale tensors – For every *.weight tensor, there must be a corresponding *.weight_scale tensor in the same checkpoint file.

The loader discovers these pairs through the get_fp8_swap_module_ops function (lines 50-61). If no valid weight/scale pairs are detected, the function raises a ValueError (lines 62-66), preventing ambiguous interpretation with BF16 checkpoints.

PyTorch Version Compatibility

Your environment must run a recent PyTorch build that supports the torch.float8_e4m3fn dtype. The FP8Linear layer constructor (lines 30-44 in fp8_scaled_mm.py) stores weights directly in this format, requiring native FP8 tensor support in your PyTorch installation.

Optional TensorRT-LLM Acceleration

While not strictly required, the implementation automatically detects and utilizes TensorRT-LLM kernels when available. The trtllm_scaled_mm_usable() function (lines 53-64) checks for this optimization. If TensorRT-LLM is unavailable, the code falls back to PyTorch's native _scaled_mm implementation (lines 66-80).

Verification and Implementation

Checking Checkpoint Compatibility

Before loading a model, verify that your checkpoint supports FP8 scaled MM quantization by attempting to discover the swap operations:

from ltx_core.quantization.fp8_scaled_mm import get_fp8_swap_module_ops

try:
    ops = get_fp8_swap_module_ops("/path/to/checkpoint.safetensors")
    print("Checkpoint is FP8-compatible – FP8Linear layers will be swapped.")
except ValueError as e:
    print("Incompatible checkpoint:", e)

This validation ensures that the required *.weight_scale tensors exist alongside the FP8 weights, preventing runtime errors during model initialization.

Loading Models with FP8 Quantization Policy

To apply FP8 scaled MM quantization, construct a quantization policy using the builder function and pass it to the model loader:

from ltx_core.quantization.fp8_scaled_mm import build_policy
from ltx_core.loader.sft_loader import load_model

checkpoint = "models/ltx-2-19b-dev-fp8.safetensors"

# Build the quantization policy

policy = build_policy(checkpoint)

# Load model with FP8 scaled MM support

model = load_model(
    checkpoint_path=checkpoint,
    quant_policy=policy,
)

The FP8Linear layers will automatically handle the scaled matrix multiplication during the forward pass, selecting the optimal kernel based on your hardware capabilities.

Alternative Path for BF16 Checkpoints

If your checkpoint contains BF16 weights without pre-calculated *.weight_scale tensors, you cannot use the scaled MM path. Instead, utilize the FP8 cast implementation available in packages/ltx-core/src/ltx_core/quantization/fp8_cast.py, which performs quantization at runtime rather than relying on pre-quantized weights.

Summary

• Pre-quantized weights required: Checkpoints must contain F8_E4M3 dtype tensors with matching *.weight_scale tensors.
• Discovery mechanism: The get_fp8_swap_module_ops function validates checkpoint compatibility by scanning for weight/scale pairs.
• PyTorch dependency: Requires a PyTorch version supporting torch.float8_e4m3fn for the FP8Linear implementation.
• Performance optimization: Automatically uses TensorRT-LLM kernels when available, falling back to PyTorch _scaled_mm otherwise.
• BF16 alternative: Use the fp8_cast.py module for checkpoints that lack pre-quantized FP8 weights.

Frequently Asked Questions

What happens if I try to use FP8 scaled MM with a BF16 checkpoint?

The get_fp8_swap_module_ops function will raise a ValueError because the absence of *.weight_scale tensors makes the quantization configuration ambiguous. For BF16 checkpoints, you must use the FP8 cast path (fp8_cast.py) instead, which handles quantization during inference rather than expecting pre-quantized weights.

Do I need TensorRT-LLM installed to use FP8 scaled MM quantization?

No, TensorRT-LLM is optional. The fp8_scaled_mm.py implementation checks trtllm_scaled_mm_usable() and automatically falls back to PyTorch's native _scaled_mm function (lines 66-80) if TensorRT-LLM is unavailable. However, having TensorRT-LLM installed provides faster inference through optimized kernels.

What is the difference between build_policy and get_fp8_swap_module_ops?

The get_fp8_swap_module_ops function performs the low-level validation, scanning the checkpoint for F8_E4M3 weights and their corresponding scale tensors (lines 50-61). The build_policy function wraps this discovery process to create a QuantizationPolicy object that the model loader in packages/ltx-core/src/ltx_core/loader/sft_loader.py can consume to initialize FP8Linear layers throughout the network.

Where are the FP8 weights stored in the model architecture?

The FP8Linear class (defined in fp8_scaled_mm.py, lines 30-44) replaces standard Linear layers when the quantization policy is applied. These layers store weights in torch.float8_e4m3fn format and apply the per-tensor scales during the forward pass using either TensorRT-LLM or PyTorch scaled matrix multiplication kernels.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}