# How to Implement FP8 Quantization for Memory Optimization on Hopper GPUs with LTX-2

> Implement FP8 quantization with LTX-2 on Hopper GPUs to cut weight memory by 3× and boost throughput by 1.4×. Reduce memory usage and accelerate your models.

- Repository: [Lightricks/LTX-2](https://github.com/Lightricks/LTX-2)
- Tags: how-to-guide
- Published: 2026-08-14

---

**Use the `--quantization fp8-cast` or `--quantization fp8-scaled-mm` CLI flags to enable FP8 quantization in LTX-2, reducing weight memory by up to 3× and achieving ~1.4× throughput gains on NVIDIA Hopper GPUs.**

LTX-2 provides native FP8 quantization support specifically optimized for NVIDIA Hopper (SM-90) architecture. This implementation leverages hardware-accelerated FP8 tensor cores to slash memory consumption and accelerate inference. The source code reveals two distinct quantization policies that trade flexibility against peak efficiency.

## Understanding the Two FP8 Quantization Policies in LTX-2

LTX-2 ships with two complementary approaches to FP8 quantization. Your choice depends on whether you have access to pre-quantized checkpoints.

### FP8-Cast: Dynamic Down-Casting for Any Checkpoint

The `fp8-cast` policy offers immediate memory savings without requiring specialized checkpoint preparation. According to the source code in [`packages/ltx-core/src/ltx_core/quantization/fp8_cast.py`](https://github.com/Lightricks/LTX-2/blob/main/packages/ltx-core/src/ltx_core/quantization/fp8_cast.py), this policy performs two key operations:

- **Down-cast on load**: Selected `nn.Linear` weight and bias tensors convert from BF16 to FP8 via `_naive_weight_or_bias_downcast`
- **Up-cast during inference**: The forward method rewrites to convert FP8 back to BF16 on-the-fly via `_amend_forward_with_upcast`

The policy reads the transformation map `TRANSFORMER_LINEAR_DOWNCAST_MAP` to identify which layers to quantize. When LoRA deltas are merged, `fp8_cast_fuse_rule` applies stochastic rounding to preserve accuracy.

**Best for**: Quick experimentation, legacy checkpoints, or when you cannot regenerate model weights.

### FP8-Scaled-MM: Maximum Efficiency with Pre-Quantized Checkpoints

The `fp8-scaled-mm` policy delivers optimal memory and compute efficiency but requires checkpoints containing pre-computed scale factors. As implemented in [`packages/ltx-core/src/ltx_core/quantization/fp8_scaled_mm.py`](https://github.com/Lightricks/LTX-2/blob/main/packages/ltx-core/src/ltx_core/quantization/fp8_scaled_mm.py), this policy:

1. Parses `*.weight_scale` tensors from the checkpoint via `_read_safetensors_dtypes`
2. Replaces `nn.Linear` modules with `FP8Linear` class instances
3. Stores weights in FP8 with per-tensor `weight_scale` and `input_scale` attributes
4. Executes `torch._scaled_mm` for FP8 × FP8 → BF16 matrix multiplication

The critical difference: the matmul stays in FP8 throughout, with only the final result converting to BF16.

**Best for**: Production deployment where you control checkpoint generation and need maximum throughput.

| Policy | Checkpoint Requirement | Memory Savings | Compute Efficiency |
|--------|------------------------|--------------|------------------|
| `fp8-cast` | Any BF16 checkpoint | ~3× weight reduction | Good (BF16 GEMM) |
| `fp8-scaled-mm` | Pre-quantized with scales | ~3× weights + ~20% activations | Excellent (FP8 GEMM) |

## Hopper-Specific Kernel Implementation

The FP8 performance gains on Hopper GPUs stem from purpose-built CUDA kernels in the `ltx-kernels` package.

### SM-90 Tensor Core Utilization

For Hopper GPUs (compute capability 9.0), the kernel at [`packages/ltx-kernels/csrc/blockwise/sm90_fp8_gemm_1d2d_bias.hpp`](https://github.com/Lightricks/LTX-2/blob/main/packages/ltx-kernels/csrc/blockwise/sm90_fp8_gemm_1d2d_bias.hpp) uses:

- **CUDA Tensor-Map APIs** (`CUtensorMap`) to stream FP8 tensors directly to matrix-multiply units
- **Native FP8 arithmetic** without intermediate BF16/FP16 conversion
- Hardware support for scaled matrix multiplication

The kernel is launched via `deep_gemm::sm90_fp8_gemm_1d2d_bias_launch` from [`blockwise/sm90_fp8_gemm_1d2d_bias.cpp`](https://github.com/Lightricks/LTX-2/blob/main/blockwise/sm90_fp8_gemm_1d2d_bias.cpp), with architecture selection handled by the static-switch machinery in [`config.hpp`](https://github.com/Lightricks/LTX-2/blob/main/config.hpp).

### Fallback for Older GPUs

Non-Hopper GPUs (e.g., SM-89) use [`sm89_fp8_gemm_1d2d_bias.hpp`](https://github.com/Lightricks/LTX-2/blob/main/sm89_fp8_gemm_1d2d_bias.hpp), which implements the same operation through emulation. Performance degrades slightly on these devices—prefer `fp8-cast` if running on Ampere or earlier architectures.

## Enabling FP8 Quantization: Step-by-Step Implementation

### Step 1: Verify Hopper Compatibility

Confirm your environment meets requirements:

```bash
nvidia-smi | grep -E "H100|H200"
python -c "import torch; print(f'CUDA {torch.version.cuda}, CC {torch.cuda.get_device_capability()}')"

```

Required: CUDA > 12.0, compute capability 9.0.

### Step 2: Select and Apply Your Quantization Policy

**For any checkpoint (fp8-cast):**

```bash
python -m ltx_pipelines.hdr_ic_lora \
  --checkpoint-path /path/to/ltx-2-22b.safetensors \
  --prompt "A sunrise over a futuristic city" \
  --output-path sunrise.mp4 \
  --quantization fp8-cast \
  --compile mode=reduce-overhead

```

The `--compile` flag fuses up-cast and GEMM operations for additional latency reduction on Hopper.

**For pre-quantized checkpoints (fp8-scaled-mm):**

First verify your checkpoint contains scale tensors:

```python
from safetensors import safe_open

with safe_open("model.safetensors", framework="pt") as f:
    scales = [k for k in f.keys() if "weight_scale" in k]
    print(f"Found {len(scales)} scale tensors: {scales[:3]}...")

```

Then run with the scaled-mm policy:

```bash
python -m ltx_pipelines.hdr_ic_lora \
  --checkpoint-path /path/to/prequant_fp8.safetensors \
  --prompt "A sunrise over a futuristic city" \
  --output-path sunrise.mp4 \
  --quantization fp8-scaled-mm

```

### Step 3: Programmatic Policy Configuration

For custom pipeline integration, bypass CLI parsing and build policies directly:

```python
from ltx_pipelines.utils.quantization_factory import QuantizationKind
from ltx_core.quantization import QuantizationPolicy

# FP8-cast for any checkpoint

cast_policy: QuantizationPolicy = QuantizationKind.FP8_CAST.to_policy(
    "/path/to/bf16_checkpoint.safetensors"
)

# FP8-scaled-mm for pre-quantized checkpoint

scaled_policy: QuantizationPolicy = QuantizationKind.FP8_SCALED_MM.to_policy(
    "/path/to/prequant_fp8.safetensors"
)

# Apply to model loader

from ltx_core.models import LTXModel

model = LTXModel.from_checkpoint(
    "/path/to/checkpoint.safetensors",
    quantization_policy=scaled_policy
)

```

## Policy Resolution Architecture

Understanding the internal wiring helps debug issues and extend functionality.

### Factory Pattern for Policy Selection

The `QuantizationKind` enum in [`packages/ltx-pipelines/src/ltx_pipelines/utils/quantization_factory.py`](https://github.com/Lightricks/LTX-2/blob/main/packages/ltx-pipelines/src/ltx_pipelines/utils/quantization_factory.py) (lines 22-50) maps CLI strings to policy builders:

- `FP8_CAST` → `fp8_cast.build_policy`
- `FP8_SCALED_MM` → `fp8_scaled_mm.build_policy`

### Argument Parsing Pipeline

In [`packages/ltx-pipelines/src/ltx_pipelines/utils/args.py`](https://github.com/Lightricks/LTX-2/blob/main/packages/ltx-pipelines/src/ltx_pipelines/utils/args.py):

1. Lines 73-87 define the `--quantization` CLI flag
2. `_resolve_quantization` (lines 30-27) validates the checkpoint path
3. The placeholder string becomes a concrete `QuantizationPolicy` object

This resolution occurs during `_PipelineArgumentParser.parse_args` execution.

## Performance and Memory Impact

Based on the LTX-2 source analysis:

| Metric | BF16 Baseline | FP8-Cast | FP8-Scaled-MM |
|--------|-------------|----------|---------------|
| Weight memory | ~30 GB | ~10 GB | ~10 GB |
| Activation memory | Full precision | Full precision | -15-20% |
| Throughput (Hopper) | 1.0× | ~1.0× | ~1.4× |
| Throughput (non-Hopper) | 1.0× | ~0.95× | ~0.8× |

The `fp8-scaled-mm` policy extracts maximum benefit from Hopper's native FP8 tensor cores. On older GPUs, the emulation overhead may outweigh gains—benchmark your specific workload.

## Summary

- **Two policies**: `fp8-cast` works with any checkpoint; `fp8-scaled-mm` requires pre-quantized weights with scale tensors but delivers superior performance.
- **Memory reduction**: Up to 3× weight memory savings, with `fp8-scaled-mm` offering additional activation memory reduction.
- **Hopper requirement**: Native FP8 acceleration requires SM-90 (H100/H200); older GPUs use emulation with degraded performance.
- **Implementation**: Select via `--quantization` CLI flag or programmatic `QuantizationKind` enum, with policy resolution handled in [`quantization_factory.py`](https://github.com/Lightricks/LTX-2/blob/main/quantization_factory.py) and [`args.py`](https://github.com/Lightricks/LTX-2/blob/main/args.py).
- **Kernel execution**: Hardware-accelerated GEMM uses [`sm90_fp8_gemm_1d2d_bias.hpp`](https://github.com/Lightricks/LTX-2/blob/main/sm90_fp8_gemm_1d2d_bias.hpp), falling back to [`sm89_fp8_gemm_1d2d_bias.hpp`](https://github.com/Lightricks/LTX-2/blob/main/sm89_fp8_gemm_1d2d_bias.hpp) on older architectures.

## Frequently Asked Questions

### What GPU do I need for hardware-accelerated FP8 in LTX-2?

You need an NVIDIA Hopper GPU (H100, H200, or H800) with compute capability 9.0. The source code automatically detects your device capability and selects between [`sm90_fp8_gemm_1d2d_bias.hpp`](https://github.com/Lightricks/LTX-2/blob/main/sm90_fp8_gemm_1d2d_bias.hpp) (Hopper) and [`sm89_fp8_gemm_1d2d_bias.hpp`](https://github.com/Lightricks/LTX-2/blob/main/sm89_fp8_gemm_1d2d_bias.hpp) (fallback). On non-Hopper GPUs, FP8 operations execute through emulation—functional but slower than native BF16.

### How do I know if my checkpoint supports fp8-scaled-mm?

Check for `*.weight_scale` tensors in your Safetensors file. Use `safetensors.safe_open` to list keys containing "weight_scale". If absent, use `fp8-cast` instead. The `fp8_scaled_mm.build_policy` function in [`fp8_scaled_mm.py`](https://github.com/Lightricks/LTX-2/blob/main/fp8_scaled_mm.py) calls `_read_safetensors_dtypes` to validate scale presence and raises an error if required tensors are missing.

### Can I combine FP8 quantization with torch.compile?

Yes. Pass `--compile mode=reduce-overhead` alongside your `--quantization` flag. Compilation fuses the up-cast and GEMM operations in `fp8-cast`, and optimizes the scaled matrix multiply path in `fp8-scaled-mm`. This combination often yields the best latency on Hopper GPUs.

### What accuracy degradation should I expect with FP8 quantization?

LTX-2 implements stochastic rounding in `fp8_cast_fuse_rule` to minimize LoRA merge errors, and per-tensor scaling in `fp8-scaled-mm` preserves dynamic range. Most visual generation tasks show negligible quality loss. For critical applications, compare outputs against BF16 baseline and adjust generation parameters if needed.