How to Use FP8 Quantization for LTX-2 GPU Memory Reduction

LTX-2 supports two FP8 quantization modes—FP8-CAST and FP8-SCALED-MM—that store model weights in 8-bit floating-point format to cut GPU memory usage by approximately 50% compared to FP16.

FP8 quantization in Lightricks/LTX-2 reduces the memory footprint of large video generation models by compressing weights to float8_e4m3fn format while maintaining inference quality. According to the LTX-2 source code, FP8-CAST performs fused up-casting to BF16 during execution, while FP8-SCALED-MM applies per-tensor scaling factors for more stable numeric behavior. Both modes require CUDA compute capability SM 89 or newer and are explicitly disabled on Apple MPS devices.

FP8 Quantization Modes Explained

LTX-2 implements two distinct approaches to FP8 quantization, selectable via configuration or command-line arguments.

FP8-CAST: Fused Up-Casting for Inference

In ltx_core/quantization/fp8_cast.py, the FP8_CAST logic (around line 327) stores weights natively in FP8 and up-casts them to BF16 on-the-fly during inference. This casting uses a fused add-round kernel that minimizes memory movement.

Typical characteristics:

• Memory savings: ~31 GiB for full-size models (~50% reduction vs. FP16)
• Compute pattern: FP8 storage → BF16 compute → FP8 gradients (if training)
• Best for: Pre-quantized checkpoints optimized for cast-based loading

FP8-SCALED-MM: Scaled Matrix Multiplication

The FP8Linear class in ltx_core/quantization/fp8_scaled_mm.py (lines 23-52) stores both FP8 weights and per-tensor or per-block scaling factors. During GEMM operations, weights are de-quantized using these scales before multiplication.

Typical characteristics:

• Memory savings: Similar to FP8-CAST (~50%)
• Numeric stability: More stable for some model architectures due to explicit scaling
• Best for: Checkpoints requiring fine-grained scale control

In ltx_pipelines/utils/quantization_factory.py (lines 23-39), the string identifiers "fp8-cast" and "fp8-scaled-mm" map to the internal QuantizationKind enum.

Enabling FP8 Quantization

Configuration via YAML

Set weight_precision in your inference configuration:

# config_fp8_cast.yaml

model:
  name: ltx-2-base
  checkpoint: /path/to/checkpoint.pt
inference:
  weight_precision: fp8-cast
  device: cuda

# config_fp8_scaled_mm.yaml

inference:
  weight_precision: fp8-scaled-mm

Command-Line Usage

The serve_captioner.py script uses FP8 by default (line 11). Override explicitly:

python -m ltx_trainer.scripts.serve_captioner \
  --checkpoint /models/ltx-2.pt \
  --weight-precision fp8-cast

CLI help text in ltx_pipelines/utils/args.py (lines 679-680) documents these options:

python -m ltx_trainer.scripts.serve_captioner --help

# Output includes: weight_precision choices: fp8, fp8-cast, fp8-scaled-mm, ...

Programmatic API

from ltx_core.quantization import QuantizationKind
from ltx_pipelines.utils.quantization_factory import build_quantizer

# Instantiate FP8-CAST quantizer

quantizer = build_quantizer(
    kind=QuantizationKind.FP8_CAST,
    checkpoint_path="models/ltx-2.pt"
)

# Apply to loaded model

model = load_my_ltx2_model()
quantizer.apply(model)

Verifying GPU Memory Reduction

Measure weight memory before and after quantization:

import torch

def measure_weight_memory(model):
    return sum(
        p.numel() * p.element_size() 
        for p in model.parameters()
    ) / 1e9

model = load_my_ltx2_model()
print(f"FP16 weight memory: {measure_weight_memory(model):.2f} GB")

# Apply FP8 quantization

from ltx_pipelines.utils.quantization_factory import build_quantizer
quantizer = build_quantizer(kind="fp8-cast")
quantizer.apply(model)

print(f"FP8 weight memory: {measure_weight_memory(model):.2f} GB")

# Expected: ~50% reduction

Hardware and Compatibility Requirements

CUDA Architecture Requirements

The ltx_kernels/blockwise/linear.py module dispatches to architecture-specific kernels:

Architecture	Compute Capability	FP8 Support
NVIDIA Ada Lovelace	SM 89	Full
NVIDIA Hopper	SM 90	Full
Pre-Ada GPUs	SM 80 and below	Not supported

Device Guard in quantization.py

The validation logic in packages/ltx-trainer/src/ltx_trainer/quantization.py (line 189) explicitly raises an error for non-CUDA devices:

# From quantization.py (line 189)

if device.type == "mps":
    raise RuntimeError("FP8 quantization is not supported on MPS devices")

When FP8 is unavailable, the system falls back to INT8 or INT4 quantization alternatives.

Checkpoint Compatibility

Some LTX-2 checkpoints are pre-quantized for specific modes. Check your checkpoint documentation:

• Checkpoints with .fp8-cast.pt suffix → Use fp8-cast
• Checkpoints with .fp8-scaled.pt suffix → Use fp8-scaled-mm
• Generic .pt checkpoints → Either mode may work; test for numeric stability

Key Source Files for FP8 Quantization

Path	Purpose
packages/ltx-trainer/src/ltx_trainer/quantization.py	Device validation and fallback logic
packages/ltx-pipelines/src/ltx_pipelines/utils/quantization_factory.py	String-to-enum mapping for FP8 modes
packages/ltx-pipelines/src/ltx_pipelines/utils/args.py	CLI argument definitions
packages/ltx-core/src/ltx_core/quantization/fp8_cast.py	FP8-CAST implementation (~line 327)
packages/ltx-core/src/ltx_core/quantization/fp8_scaled_mm.py	FP8Linear and scaled MM (lines 23-52)
packages/ltx-kernels/src/ltx_kernels/blockwise/linear.py	CUDA kernel dispatch for SM 89/90

Summary

• FP8-CAST stores weights in float8_e4m3fn and up-casts to BF16 during inference via fused kernels in fp8_cast.py
• FP8-SCALED-MM adds per-tensor scaling factors for de-quantization during GEMM, implemented in fp8_scaled_mm.py
• Both modes reduce GPU memory by ~50% compared to FP16, from ~62 GiB to ~31 GiB for full LTX-2 models
• Activation via weight_precision: fp8-cast or fp8-scaled-mm in YAML configs, or --weight-precision CLI flag
• Requires CUDA SM 89+ (Ada Lovelace/Hopper); MPS devices are blocked by explicit guard in quantization.py

Frequently Asked Questions

What GPUs support FP8 quantization in LTX-2?

NVIDIA Ada Lovelace (RTX 40-series, SM 89) and Hopper (H100, SM 90) GPUs support FP8 natively. The ltx_kernels/blockwise/linear.py module dispatches architecture-specific kernels based on detected compute capability. Older GPUs (Ampere and earlier) lack native FP8 hardware support and will raise errors or fall back to INT quantization.

How do I choose between FP8-CAST and FP8-SCALED-MM?

Select FP8-CAST for checkpoints pre-optimized for cast-based loading and when maximum inference speed is prioritized. Choose FP8-SCALED-MM when your checkpoint includes scale factors or when you observe numeric instability with pure casting. The quantization_factory.py module validates that your selected mode matches checkpoint requirements.

Can I train with FP8 quantization enabled?

The source code shows FP8 is primarily implemented for inference weight storage, with training.precision typically remaining at FP16 or BF16 for gradient computation. The FP8Linear modules handle weight gradients carefully—check fp8_scaled_mm.py for training-specific behavior if fine-tuning quantized models.

Why does FP8 quantization fail on my Mac?

The guard clause in quantization.py (line 189) explicitly raises RuntimeError for MPS devices because Apple's Metal Performance Shaders lack native FP8 support. Use INT8 or INT4 quantization alternatives, or run on CUDA hardware.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}