# How Stable Diffusion's Latent Diffusion Architecture Differs from Pixel-Space Models

> Explore how Stable Diffusion's latent diffusion architecture outperforms pixel-space models by moving computation to a compressed latent space, significantly reducing resource needs without sacrificing quality.

- Repository: [CompVis - Computer Vision and Learning LMU Munich/stable-diffusion](https://github.com/CompVis/stable-diffusion)
- Tags: architecture
- Published: 2026-03-01

---

**Stable Diffusion moves the computationally expensive diffusion process from high-dimensional pixel space into a compressed latent space using a variational autoencoder, reducing memory and compute requirements by approximately 64× while maintaining generation quality through a separate encoder-decoder pipeline.**

The **CompVis/stable-diffusion** repository implements a **latent diffusion model (LDM)** that fundamentally departs from traditional pixel-space diffusion by operating on compressed latent representations rather than raw RGB values. This architectural shift enables high-resolution image generation on consumer hardware while preserving the quality benchmarks of much larger pixel-space models. Understanding this two-stage pipeline is essential for optimizing inference speed and training custom diffusion models.

## The Two-Stage Latent Diffusion Pipeline

### Stage 1: Autoencoder-Based Compression

In [`ldm/models/autoencoder.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/models/autoencoder.py), the `AutoencoderKL` class implements the encoder-decoder pair that compresses full-resolution images (e.g., 512×512×3 pixels) into compact latent tensors (e.g., 64×64×4). The `Encoder` transforms input images into latent representations `z`, while the `Decoder` reconstructs images from these latents during final output generation.

This architecture employs a down-sampling factor **f = 8**, yielding an effective 64-dimensional reduction in data volume compared to operating directly on pixel arrays. The model card in [`Stable_Diffusion_v1_Model_Card.md`](https://github.com/CompVis/stable-diffusion/blob/main/Stable_Diffusion_v1_Model_Card.md) explicitly describes this design as combining "an autoencoder with a diffusion model that is trained in the latent space of the autoencoder."

### Stage 2: Diffusion in Latent Space

Instead of adding noise to pixels, the diffusion process operates directly on the 4-channel latent tensors. The `UNetModel` class in [`ldm/modules/diffusionmodules/openaimodel.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/modules/diffusionmodules/openaimodel.py) implements the denoising backbone, accepting latent inputs rather than 3-channel RGB images.

This allows the same UNet architecture used in pixel-space diffusion (OpenAI's ADM) to process significantly smaller tensors. The input channels correspond to the latent dimension (typically 4) rather than RGB channels (3), reducing GPU memory consumption during both training and inference while maintaining the same cross-attention mechanisms for text conditioning.

## Key Architectural Differences from Pixel-Space Diffusion

**Input Dimensionality**: Pixel-space models like DDPM operate on full-resolution RGB tensors (3×H×W), whereas Stable Diffusion processes 4×(H/8)×(W/8) latent tensors. This dimensional reduction fundamentally changes the computational graph's memory footprint.

**Computational Efficiency**: Latent tensors are approximately 64× smaller than their pixel equivalents. The 860-million-parameter UNet in Stable Diffusion can train and infer at speeds impossible for pixel-space models of equivalent capacity, as implemented in the [`openaimodel.py`](https://github.com/CompVis/stable-diffusion/blob/main/openaimodel.py) source.

**Memory Requirements**: Operating on 64×64×4 tensors instead of 512×512×3 reduces activation memory by two orders of magnitude. This efficiency enables the `UNetModel` to run high-resolution generation on consumer GPUs with 8-16GB VRAM.

**Quality Preservation**: The autoencoder learns a lossy but perceptually faithful compression in [`ldm/models/autoencoder.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/models/autoencoder.py). The diffusion model generates plausible latents, and the decoder recovers high-frequency details without the UNet ever processing raw pixels.

**Modularity**: The encoder-decoder pair defined in [`autoencoder.py`](https://github.com/CompVis/stable-diffusion/blob/main/autoencoder.py) can be swapped (e.g., for VQ-GAN variants) without retraining the core diffusion UNet. This flexibility allows adapting the pipeline to different data modalities while reusing the same latent diffusion weights.

## Training and Inference Implementation

During training, the pipeline follows a strict sequence as implemented in the source:

1. **Encode**: Input images pass through `AutoencoderKL.encode` to produce latent representations `z`.
2. **Add Noise**: Gaussian noise is applied to `z` following the standard diffusion schedule defined in the training configuration.
3. **Denoise**: The `UNetModel` predicts noise (or clean latents) conditioned on text embeddings from CLIP, operating entirely within the latent domain via the forward pass in [`openaimodel.py`](https://github.com/CompVis/stable-diffusion/blob/main/openaimodel.py).
4. **Compute Loss**: The `training_step` method in [`autoencoder.py`](https://github.com/CompVis/stable-diffusion/blob/main/autoencoder.py) calculates reconstruction loss between predicted and original latents.
5. **Decode**: For visualization, `AutoencoderKL.decode` reconstructs the final image from denoised latents.

## Running Stable Diffusion: Code Examples

### Text-to-Image Generation

```bash
python scripts/txt2img.py \
  --prompt "a photograph of an astronaut riding a horse" \
  --plms \
  --ckpt models/ldm/stable-diffusion-v1/model.ckpt \
  --outdir samples/

```

This script orchestrates the full latent pipeline: text encoding, latent denoising via `UNetModel`, and final image reconstruction through the autoencoder decoder defined in [`ldm/models/autoencoder.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/models/autoencoder.py).

### Image-to-Image Editing

```bash
python scripts/img2img.py \
  --init-img path/to/input.jpg \
  --prompt "A fantasy landscape, trending on artstation" \
  --strength 0.8 \
  --ckpt models/ldm/stable-diffusion-v1/model.ckpt

```

The [`img2img.py`](https://github.com/CompVis/stable-diffusion/blob/main/img2img.py) script encodes the input image into latents using the encoder, applies partial noise according to the strength parameter, then runs the latent diffusion denoising process before decoding back to pixel space.

### Python API Usage

```python
from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_pretrained(
    "CompVis/stable-diffusion-v1-4",
    torch_dtype="float16"
).to("cuda")

image = pipe("a photo of an astronaut riding a horse on mars")["images"][0]
image.save("astronaut.png")

```

Under the hood, this loads the identical encoder, `UNetModel`, and decoder weights from the CompVis repository, managing the latent space transitions transparently.

## Summary

- Stable Diffusion implements **latent diffusion models (LDMs)**, separating generation into autoencoder compression and latent-space denoising stages.
- The architecture reduces computational complexity by operating on 64×64×4 tensors rather than full-resolution pixels, achieving approximately 64× efficiency gains over pixel-space approaches.
- Core components reside in [`ldm/models/autoencoder.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/models/autoencoder.py) (encoder/decoder pair) and [`ldm/modules/diffusionmodules/openaimodel.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/modules/diffusionmodules/openaimodel.py) (latent UNet).
- This design maintains generation quality while enabling training and inference on consumer hardware, distinguishing it fundamentally from pixel-space diffusion models like DDPM or ADM.

## Frequently Asked Questions

### Why does Stable Diffusion use latent space instead of pixel space?

Operating in latent space reduces the dimensionality of data processed by the diffusion UNet by a factor of 64, dramatically lowering GPU memory requirements and computational cost. The autoencoder handles perceptual compression and reconstruction in [`ldm/models/autoencoder.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/models/autoencoder.py), allowing the diffusion model to focus on semantic generation within a compact representation.

### What is the down-sampling factor in Stable Diffusion's autoencoder?

The default configuration uses a down-sampling factor **f = 8**, converting a 512×512×3 pixel image into a 64×64×4 latent tensor. This 8× spatial reduction in both height and width creates the 64× total compression factor that defines the latent diffusion architecture's efficiency advantage.

### Can the autoencoder be replaced without retraining the diffusion UNet?

Yes. The modular design in [`ldm/models/autoencoder.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/models/autoencoder.py) allows swapping the encoder-decoder pair (e.g., switching from KL-regularized autoencoders to VQ-GAN variants) while retaining the same `UNetModel` in [`openaimodel.py`](https://github.com/CompVis/stable-diffusion/blob/main/openaimodel.py). This flexibility enables adapting the pipeline to different data modalities without modifying the core latent diffusion weights.

### How does conditioning work in latent diffusion compared to pixel-space models?

Both architectures use cross-attention mechanisms for text guidance, but in Stable Diffusion these operations occur on latent features rather than pixel features. Because the UNet in [`ldm/modules/diffusionmodules/openaimodel.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/modules/diffusionmodules/openaimodel.py) processes 4-channel latent tensors instead of 3-channel RGB images, the attention computations require significantly less memory while maintaining equivalent conditioning fidelity through the same attention layers.