How to Implement Mixed Precision (FP16) Inference in Stable Diffusion for Faster Generation

Use the --precision autocast flag in the official CompVis/stable-diffusion scripts to enable automatic mixed precision, or convert model weights to FP16 during loading to halve VRAM usage and accelerate inference by up to 50%.

Mixed precision (FP16) inference reduces memory bandwidth and leverages Tensor Cores on NVIDIA GPUs to generate images faster without sacrificing quality. The CompVis/stable-diffusion repository provides built-in support for automatic mixed precision through torch.autocast, while also allowing advanced users to force FP16 weights for maximum efficiency. This guide covers both approaches using the actual source code implementation.

How torch.autocast Enables Mixed Precision

The inference pipeline in scripts/txt2img.py uses PyTorch’s automatic mixed precision context manager to handle FP16 conversion dynamically.

The Precision Scope Context

In [scripts/txt2img.py](https://github.com/CompVis/stable-diffusion/blob/main/scripts/txt2img.py#L14-L16), the script imports both autocast and nullcontext:

from torch import autocast
from contextlib import nullcontext

At line 288, the script selects the appropriate context based on the --precision argument:

precision_scope = autocast if opt.precision == "autocast" else nullcontext

When autocast is active, CUDA kernels automatically cast compatible operations (like matrix multiplications) to FP16 while keeping numerically sensitive operations in FP32. This happens inside the sampling loop without requiring changes to the model architecture or sampler code in [ldm/models/diffusion/ddim.py](https://github.com/CompVis/stable-diffusion/blob/main/ldm/models/diffusion/ddim.py).

Memory and Speed Benefits

Reduced VRAM consumption occurs because FP16 tensors occupy half the memory of FP32, allowing larger batch sizes or higher resolution generation on consumer GPUs. Tensor Core acceleration provides 2×–8× speedup for compatible operations by utilizing NVIDIA’s mixed-precision hardware units.

Enabling FP16 Inference via Command Line

The simplest method requires no code changes—just add the precision flag when running the standard inference scripts.

Text-to-Image with Autocast

Run the following command to enable mixed precision for txt2img generation:

python scripts/txt2img.py \
    --prompt "A beautiful sunrise over mountains" \
    --ckpt_path models/ldm/stable-diffusion-v1/model.ckpt \
    --precision autocast \
    --n_samples 4 \
    --ddim_steps 50

The --precision autocast argument triggers the precision_scope context manager at line 288 of [scripts/txt2img.py](https://github.com/CompVis/stable-diffusion/blob/main/scripts/txt2img.py), wrapping the sampling loop in torch.autocast("cuda"). The same flag works identically in [scripts/img2img.py](https://github.com/CompVis/stable-diffusion/blob/main/scripts/img2img.py) for image-to-image generation.

Converting Model Weights to FP16 for Maximum Performance

For advanced scenarios, you can convert the entire model to FP16 once during loading rather than relying on per-operation autocasting. This eliminates overhead and maximizes VRAM savings.

Custom FP16 Model Loader

Replace the standard load_model_from_config call in [scripts/txt2img.py](https://github.com/CompVis/stable-diffusion/blob/main/scripts/txt2img.py#L55-L59) with this helper function:

import torch
from omegaconf import OmegaConf
from ldm.util import instantiate_from_config

def load_model_fp16(config_path: str, ckpt_path: str, device: str = "cuda"):
    """
    Load Stable Diffusion checkpoint and convert all weights to FP16.
    """
    config = OmegaConf.load(config_path)
    pl_sd = torch.load(ckpt_path, map_location="cpu")
    sd = pl_sd["state_dict"]
    
    # Convert every parameter to half-precision before GPU transfer

    for k in sd:
        sd[k] = sd[k].half()
    
    model = instantiate_from_config(config.model)
    model.load_state_dict(sd, strict=False)
    model.to(device)  # FP16 tensors on GPU

    model.eval()
    return model

Then modify the loading call:

model = load_model_fp16(opt.config, opt.ckpt, device)

Key differences from autocast:

• All parameters reside permanently in torch.float16, reducing VRAM usage by approximately 50%.
• You can use nullcontext instead of autocast since the tensors are already FP16.
• Warning: Some operations (batch normalization, certain activations) may become numerically unstable in pure FP16.

Safest Implementation Strategy

Combine both approaches for optimal stability and performance. Convert weights to FP16 for memory savings, but retain the autocast context to protect numerically sensitive operations:

with torch.autocast("cuda"):
    samples, _ = sampler.sample(S=50,
                                batch_size=4,
                                shape=(4, 64, 64),
                                conditioning=cond,
                                verbose=False)

This configuration runs the model weights in FP16 while allowing PyTorch to automatically upcast specific operations to FP32 where necessary.

Performance Optimization Checklist

Follow these steps to maximize inference speed and minimize memory usage:

1. Enable cuDNN benchmarking before model loading to optimize kernel selection:

   torch.backends.cudnn.benchmark = True

2. Start with the CLI flag (--precision autocast) for an immediate 30–50% speedup with approximately 2GB VRAM savings at 512×512 resolution.

3. Convert weights to FP16 using the custom loader for an additional 10–20% speedup and further VRAM reduction, enabling larger batch sizes.

4. Verify output quality by comparing FP16 samples against full-precision runs; FP16 should produce visually indistinguishable results for most prompts, though rare numerical artifacts may require falling back to autocast-only mode.

Summary

• Automatic mixed precision is available out-of-the-box via --precision autocast in scripts/txt2img.py and scripts/img2img.py, utilizing torch.autocast to dynamically cast operations to FP16.
• Weight conversion to FP16 during model loading halves VRAM usage by calling .half() on state dict tensors before model.to(device).
• Sampler agnostic implementation means DDIMSampler, PLMSSampler, and DPMSolverSampler in ldm/models/diffusion/ require no modifications to support mixed precision.
• Combined approach of FP16 weights plus autocast context provides the best balance of speed, memory efficiency, and numerical stability.

Frequently Asked Questions

Does mixed precision inference reduce image quality in Stable Diffusion?

No, FP16 inference produces visually identical results to FP32 for most prompts and seeds. The torch.autocast context automatically keeps sensitive operations (like softmax and logarithms) in FP32 to prevent numerical instability. If you notice artifacts with pure FP16 weights, use the autocast wrapper to maintain quality while preserving most performance gains.

Which GPUs benefit most from FP16 inference?

NVIDIA GPUs with Tensor Cores (RTX 20 series and newer, Tesla V100/A100/H100) see the largest speedups—often 2× faster than FP32. Older Pascal architecture cards (GTX 10 series) support FP16 but lack Tensor Cores, providing only memory bandwidth benefits rather than computational acceleration. Always ensure your CUDA version supports the torch.autocast API (PyTorch 1.6+).

Can I use mixed precision with the img2img script?

Yes, the --precision autocast flag works identically in [scripts/img2img.py](https://github.com/CompVis/stable-diffusion/blob/main/scripts/img2img.py) because both scripts share the same precision_scope logic at line 288. The DDIM sampler and other diffusion samplers in ldm/models/diffusion/ are dtype-agnostic and operate correctly under autocast regardless of whether you are performing text-to-image or image-to-image generation.

How much VRAM does FP16 inference save?

Converting to FP16 reduces model weight memory consumption by approximately 50%, typically saving 1.5–2.5GB for the standard Stable Diffusion v1.4/v1.5 checkpoint. This allows you to increase batch sizes from 1 to 2-4 on 8GB consumer GPUs, or generate at 768×768 resolution where 512×512 previously exhausted memory. Additional savings come from activation checkpointing when using torch.autocast.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}