# VAE Autoencoder Architecture in Stable Diffusion: How Latent Space Compression Works

> Explore the VAE autoencoder architecture in Stable Diffusion. Learn how this powerful model compresses images into a compact latent space for efficient processing and generation.

- Repository: [CompVis - Computer Vision and Learning LMU Munich/stable-diffusion](https://github.com/CompVis/stable-diffusion)
- Tags: architecture
- Published: 2026-03-02

---

**The Stable Diffusion VAE autoencoder compresses 256×256 RGB images into a compact 16×16×64 latent representation using a hierarchical Encoder-Decoder architecture with residual blocks and attention, achieving approximately 48× compression through four stages of stride-2 down-sampling and diagonal Gaussian parameterization.**

The variational autoencoder (VAE) serves as the critical first stage of the Stable Diffusion pipeline, transforming high-resolution pixel space into a compressed latent space where the diffusion model operates efficiently. Implemented in the CompVis/stable-diffusion repository as the `AutoencoderKL` class, this U-Net-style architecture dramatically reduces computational requirements by encoding images into a lower-dimensional representation while preserving visual fidelity through a symmetric encoder and decoder structure.

## Core Architecture Components

The VAE implementation consists of two primary building blocks defined in [`ldm/modules/diffusionmodules/model.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/modules/diffusionmodules/model.py), wrapped by a high-level interface in [`ldm/models/autoencoder.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/models/autoencoder.py):

- **Encoder**: Processes input images through successive down-sampling stages, applying residual blocks and attention to produce the parameters of a diagonal Gaussian distribution.
- **Decoder**: Mirrors the encoder architecture, up-sampling latent representations back to full-resolution images using residual blocks and attention layers.

Both components share configuration parameters defined in the `ddconfig` section of YAML files such as [`configs/autoencoder/autoencoder_kl_8x8x64.yaml`](https://github.com/CompVis/stable-diffusion/blob/main/configs/autoencoder/autoencoder_kl_8x8x64.yaml), specifying channel multipliers, resolution targets, and attention placements.

## Encoder: Hierarchical Down-Sampling to Latent Space

The `Encoder` class in [`ldm/modules/diffusionmodules/model.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/modules/diffusionmodules/model.py) implements a multi-resolution feature extraction pipeline. It begins with an initial 3×3 convolution (`self.conv_in`) that projects the input image into the base channel dimension.

The architecture processes data through multiple resolution levels determined by the `ch_mult` parameter. For the default configuration with `ch_mult = [1,1,2,2,4,4]`, the encoder performs **four down-sampling steps**, each implemented via stride-2 convolutions in the `Downsample` class:

```python

# From ldm/modules/diffusionmodules/model.py

if i_level != self.num_resolutions-1:
    downsample = Downsample(block_in, resamp_with_conv)

```

At each resolution level, the encoder stacks `num_res_blocks` instances of `ResnetBlock`, which provide two 3×3 convolutions with GroupNorm and Swish activation, plus optional dropout. When the current resolution matches entries in `attn_resolutions` (typically 16×16 and 8×8), self-attention layers (`AttnBlock` or `LinearAttention`) are inserted to capture long-range dependencies.

The final output projection uses a 3×3 convolution producing `2*z_channels` when `double_z=True`, representing the mean and log-variance parameters stacked along the channel dimension:

```python
self.conv_out = torch.nn.Conv2d(block_in,
                                2*z_channels if double_z else z_channels,
                                kernel_size=3, stride=1, padding=1)

```

## Decoder: Reconstructing Images from Latents

The `Decoder` class mirrors the encoder's hierarchical structure but operates in reverse, up-sampling the latent representation back to pixel space. It accepts a tensor of shape `[B, z_channels, H/16, W/16]` and processes it through a series of residual blocks and attention layers at decreasing spatial resolutions.

Up-sampling occurs via the `Upsample` class, which uses nearest-neighbor interpolation with a scale factor of 2 followed by an optional 3×3 convolution. The decoder inserts attention blocks at the same resolutions specified in `attn_resolutions`, ensuring symmetry with the encoder's computational path.

After processing through the final resolution level, a 3×3 convolution maps the features to the target output channels (typically 3 for RGB). Optional `tanh_out` activation can be applied for specific training regimes.

## How Latent Space Compression Works

The compression mechanism follows a variational Bayesian approach implemented in the `AutoencoderKL` wrapper class:

1. **Forward Pass**: The encoder receives an image tensor **x** with shape `[B, 3, H, W]` (e.g., 256×256) and outputs a tensor of shape `[B, 2*z_channels, H/16, W/16]`.
2. **Distribution Parameterization**: The `AutoencoderKL` splits the encoder output into mean (**μ**) and log-variance (**logσ²**) tensors, each with `z_channels` dimensions.
3. **Sampling**: The system creates a `DiagonalGaussianDistribution` from these parameters. During training, it samples **z = μ + σ·ε** where **ε ~ N(0, I)**; during inference, it uses the deterministic mode **μ**.
4. **Compression Ratio**: With default settings (`z_channels=64`, 256×256 input, 16×16 latent spatial size), the representation compresses the image by approximately **48×**, reducing memory and compute requirements for the subsequent diffusion process.

## Key Configuration Parameters

The VAE behavior is controlled via the `ddconfig` dictionary in the autoencoder YAML configuration:

- **`ch_mult`**: Channel multipliers defining the number of resolution levels and feature dimensions at each stage.
- **`num_res_blocks`**: Number of residual blocks applied at each resolution level.
- **`attn_resolutions`**: List of spatial resolutions (as integers) where attention layers are inserted.
- **`z_channels`**: Dimensionality of the latent space per spatial location (default 64).
- **`double_z`**: Boolean flag determining whether the encoder outputs both mean and log-variance (True) or just the latent (False).

## Practical Code Example: Encoding and Decoding Images

The following example demonstrates how to instantiate the `AutoencoderKL` and perform latent compression:

```python
import torch
from omegaconf import OmegaConf
from ldm.util import instantiate_from_config
from ldm.models.autoencoder import AutoencoderKL

# Load configuration

conf_path = "configs/autoencoder/autoencoder_kl_8x8x64.yaml"
cfg = OmegaConf.load(conf_path)

# Initialize model

model: AutoencoderKL = instantiate_from_config(cfg.model)

# Load pretrained weights

ckpt = torch.load("sd-v1-4.ckpt", map_location="cpu")["state_dict"]
model.load_state_dict(ckpt, strict=False)
model.eval()

# Encode image to latent space

img = torch.randn(1, 3, 256, 256)  # Replace with actual image tensor

posterior = model.encode(img)       # Returns DiagonalGaussianDistribution

z = posterior.sample()             # Stochastic latent [1, 64, 16, 16]

# Decode back to pixels

recon = model.decode(z)              # [1, 3, 256, 256]

# Deterministic latent for inference

z_det = posterior.mode()             # Same shape, no randomness

```

## Summary

- The Stable Diffusion VAE uses a **symmetric Encoder-Decoder architecture** implemented in [`ldm/modules/diffusionmodules/model.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/modules/diffusionmodules/model.py) and wrapped by `AutoencoderKL` in [`ldm/models/autoencoder.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/models/autoencoder.py).
- **Four stages of stride-2 down-sampling** reduce spatial resolution from 256×256 to 16×16, while residual blocks and attention layers preserve feature fidelity.
- The latent space follows a **diagonal Gaussian parameterization**, outputting mean and log-variance tensors of shape `[B, z_channels, H/16, W/16]` for configurable `z_channels` (default 64).
- This architecture achieves approximately **48× compression**, enabling the diffusion model to operate efficiently in latent space rather than high-resolution pixel space.

## Frequently Asked Questions

### What is the compression factor of the Stable Diffusion VAE?

The default VAE configuration achieves approximately **48× compression**, reducing a 256×256×3 pixel image to a 16×16×64 latent representation. This factor balances reconstruction quality with computational efficiency for the diffusion process.

### How does the VAE produce stochastic versus deterministic latents?

The encoder outputs parameters for a `DiagonalGaussianDistribution`. Calling `posterior.sample()` draws a random latent **z** using the reparameterization trick (**z = μ + σ·ε**), while `posterior.mode()` returns the deterministic mean **μ** alone. Training uses sampling for regularization; inference typically uses the mode for consistency.

### Where are the attention layers placed in the VAE architecture?

Self-attention blocks are inserted at specific resolutions defined by the `attn_resolutions` configuration parameter, typically at 16×16 and 8×8 spatial sizes during the down-sampling path. These layers enable the model to capture global dependencies that convolutions miss at low resolutions.

### Can the VAE latent dimensions be changed from the default 16×16×64?

Yes. The latent shape is fully configurable via the `ddconfig` section in YAML files like [`configs/autoencoder/autoencoder_kl_8x8x64.yaml`](https://github.com/CompVis/stable-diffusion/blob/main/configs/autoencoder/autoencoder_kl_8x8x64.yaml). Adjusting `ch_mult`, `num_res_blocks`, and `z_channels` modifies the spatial compression factor and channel depth, though pretrained Stable Diffusion checkpoints require the specific configuration used during training.