VAE Autoencoder Architecture in Stable Diffusion: How Latent Space Compression Works

The Stable Diffusion VAE autoencoder compresses 256×256 RGB images into a compact 16×16×64 latent representation using a hierarchical Encoder-Decoder architecture with residual blocks and attention, achieving approximately 48× compression through four stages of stride-2 down-sampling and diagonal Gaussian parameterization.

The variational autoencoder (VAE) serves as the critical first stage of the Stable Diffusion pipeline, transforming high-resolution pixel space into a compressed latent space where the diffusion model operates efficiently. Implemented in the CompVis/stable-diffusion repository as the AutoencoderKL class, this U-Net-style architecture dramatically reduces computational requirements by encoding images into a lower-dimensional representation while preserving visual fidelity through a symmetric encoder and decoder structure.

Core Architecture Components

The VAE implementation consists of two primary building blocks defined in ldm/modules/diffusionmodules/model.py, wrapped by a high-level interface in ldm/models/autoencoder.py:

• Encoder: Processes input images through successive down-sampling stages, applying residual blocks and attention to produce the parameters of a diagonal Gaussian distribution.
• Decoder: Mirrors the encoder architecture, up-sampling latent representations back to full-resolution images using residual blocks and attention layers.

Both components share configuration parameters defined in the ddconfig section of YAML files such as configs/autoencoder/autoencoder_kl_8x8x64.yaml, specifying channel multipliers, resolution targets, and attention placements.

Encoder: Hierarchical Down-Sampling to Latent Space

The Encoder class in ldm/modules/diffusionmodules/model.py implements a multi-resolution feature extraction pipeline. It begins with an initial 3×3 convolution (self.conv_in) that projects the input image into the base channel dimension.

The architecture processes data through multiple resolution levels determined by the ch_mult parameter. For the default configuration with ch_mult = [1,1,2,2,4,4], the encoder performs four down-sampling steps, each implemented via stride-2 convolutions in the Downsample class:

# From ldm/modules/diffusionmodules/model.py

if i_level != self.num_resolutions-1:
    downsample = Downsample(block_in, resamp_with_conv)

At each resolution level, the encoder stacks num_res_blocks instances of ResnetBlock, which provide two 3×3 convolutions with GroupNorm and Swish activation, plus optional dropout. When the current resolution matches entries in attn_resolutions (typically 16×16 and 8×8), self-attention layers (AttnBlock or LinearAttention) are inserted to capture long-range dependencies.

The final output projection uses a 3×3 convolution producing 2*z_channels when double_z=True, representing the mean and log-variance parameters stacked along the channel dimension:

self.conv_out = torch.nn.Conv2d(block_in,
                                2*z_channels if double_z else z_channels,
                                kernel_size=3, stride=1, padding=1)

Decoder: Reconstructing Images from Latents

The Decoder class mirrors the encoder's hierarchical structure but operates in reverse, up-sampling the latent representation back to pixel space. It accepts a tensor of shape [B, z_channels, H/16, W/16] and processes it through a series of residual blocks and attention layers at decreasing spatial resolutions.

Up-sampling occurs via the Upsample class, which uses nearest-neighbor interpolation with a scale factor of 2 followed by an optional 3×3 convolution. The decoder inserts attention blocks at the same resolutions specified in attn_resolutions, ensuring symmetry with the encoder's computational path.

After processing through the final resolution level, a 3×3 convolution maps the features to the target output channels (typically 3 for RGB). Optional tanh_out activation can be applied for specific training regimes.

How Latent Space Compression Works

The compression mechanism follows a variational Bayesian approach implemented in the AutoencoderKL wrapper class:

1. Forward Pass: The encoder receives an image tensor x with shape [B, 3, H, W] (e.g., 256×256) and outputs a tensor of shape [B, 2*z_channels, H/16, W/16].
2. Distribution Parameterization: The AutoencoderKL splits the encoder output into mean (μ) and log-variance (logσ²) tensors, each with z_channels dimensions.
3. Sampling: The system creates a DiagonalGaussianDistribution from these parameters. During training, it samples z = μ + σ·ε where ε ~ N(0, I); during inference, it uses the deterministic mode μ.
4. Compression Ratio: With default settings (z_channels=64, 256×256 input, 16×16 latent spatial size), the representation compresses the image by approximately 48×, reducing memory and compute requirements for the subsequent diffusion process.

Key Configuration Parameters

The VAE behavior is controlled via the ddconfig dictionary in the autoencoder YAML configuration:

• ch_mult: Channel multipliers defining the number of resolution levels and feature dimensions at each stage.
• num_res_blocks: Number of residual blocks applied at each resolution level.
• attn_resolutions: List of spatial resolutions (as integers) where attention layers are inserted.
• z_channels: Dimensionality of the latent space per spatial location (default 64).
• double_z: Boolean flag determining whether the encoder outputs both mean and log-variance (True) or just the latent (False).

Practical Code Example: Encoding and Decoding Images

The following example demonstrates how to instantiate the AutoencoderKL and perform latent compression:

import torch
from omegaconf import OmegaConf
from ldm.util import instantiate_from_config
from ldm.models.autoencoder import AutoencoderKL

# Load configuration

conf_path = "configs/autoencoder/autoencoder_kl_8x8x64.yaml"
cfg = OmegaConf.load(conf_path)

# Initialize model

model: AutoencoderKL = instantiate_from_config(cfg.model)

# Load pretrained weights

ckpt = torch.load("sd-v1-4.ckpt", map_location="cpu")["state_dict"]
model.load_state_dict(ckpt, strict=False)
model.eval()

# Encode image to latent space

img = torch.randn(1, 3, 256, 256)  # Replace with actual image tensor

posterior = model.encode(img)       # Returns DiagonalGaussianDistribution

z = posterior.sample()             # Stochastic latent [1, 64, 16, 16]

# Decode back to pixels

recon = model.decode(z)              # [1, 3, 256, 256]

# Deterministic latent for inference

z_det = posterior.mode()             # Same shape, no randomness

Summary

• The Stable Diffusion VAE uses a symmetric Encoder-Decoder architecture implemented in ldm/modules/diffusionmodules/model.py and wrapped by AutoencoderKL in ldm/models/autoencoder.py.
• Four stages of stride-2 down-sampling reduce spatial resolution from 256×256 to 16×16, while residual blocks and attention layers preserve feature fidelity.
• The latent space follows a diagonal Gaussian parameterization, outputting mean and log-variance tensors of shape [B, z_channels, H/16, W/16] for configurable z_channels (default 64).
• This architecture achieves approximately 48× compression, enabling the diffusion model to operate efficiently in latent space rather than high-resolution pixel space.

Frequently Asked Questions

What is the compression factor of the Stable Diffusion VAE?

The default VAE configuration achieves approximately 48× compression, reducing a 256×256×3 pixel image to a 16×16×64 latent representation. This factor balances reconstruction quality with computational efficiency for the diffusion process.

How does the VAE produce stochastic versus deterministic latents?

The encoder outputs parameters for a DiagonalGaussianDistribution. Calling posterior.sample() draws a random latent z using the reparameterization trick (z = μ + σ·ε), while posterior.mode() returns the deterministic mean μ alone. Training uses sampling for regularization; inference typically uses the mode for consistency.

Where are the attention layers placed in the VAE architecture?

Self-attention blocks are inserted at specific resolutions defined by the attn_resolutions configuration parameter, typically at 16×16 and 8×8 spatial sizes during the down-sampling path. These layers enable the model to capture global dependencies that convolutions miss at low resolutions.

Can the VAE latent dimensions be changed from the default 16×16×64?

Yes. The latent shape is fully configurable via the ddconfig section in YAML files like configs/autoencoder/autoencoder_kl_8x8x64.yaml. Adjusting ch_mult, num_res_blocks, and z_channels modifies the spatial compression factor and channel depth, though pretrained Stable Diffusion checkpoints require the specific configuration used during training.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}