How CLIP ViT-L/14 Text Encoder Conditions the UNet in Stable Diffusion: Cross-Attention Deep Dive

The CLIP ViT-L/14 text encoder transforms text prompts into 768-dimensional latent conditioning vectors that guide the UNet denoiser through cross-attention layers, enabling semantic text-to-image generation.

Stable Diffusion relies on a frozen CLIP ViT-L/14 text encoder to bridge natural language and latent image space. According to the CompVis/stable-diffusion source code, text embeddings are injected into the UNet architecture via specialized cross-attention mechanisms that preserve spatial structure while incorporating semantic information from the prompt.

Generating Latent Conditionings with FrozenCLIPTextEmbedder

The conditioning pipeline begins in ldm/modules/encoders/modules.py with the FrozenCLIPTextEmbedder class. This wrapper loads a frozen CLIP ViT-L/14 model and converts raw text into numerical embeddings suitable for the diffusion process.

class FrozenCLIPTextEmbedder(nn.Module):
    def __init__(self, version='ViT-L/14', device="cuda", max_length=77,
                 n_repeat=1, normalize=True):
        self.model, _ = clip.load(version, jit=False, device="cpu")
    
    def forward(self, text):
        tokens = clip.tokenize(text).to(self.device)          # → [B, 77]

        z = self.model.encode_text(tokens)                    # → [B, 768]

        if self.normalize:
            z = z / torch.linalg.norm(z, dim=1, keepdim=True)
        return z
    
    def encode(self, text):
        z = self(text)                                        # [B, 768]

        if z.ndim == 2:
            z = z[:, None, :]                                 # → [B, 1, 768]

        z = repeat(z, 'b 1 d -> b k d', k=self.n_repeat)     # → [B, K, 768]

        return z

Key implementation details include:

• Frozen weights: The encoder operates in eval mode with gradients disabled, ensuring the CLIP parameters remain constant during diffusion training.
• Tokenization: Text is tokenized to a fixed 77-token sequence (clip.tokenize), truncating or padding as necessary.
• Output shape: The encode() method returns a tensor of shape [batch_size, context_length, 768], where 768 matches the ViT-L/14 embedding dimension.
• Repetition: The n_repeat parameter (default 1) repeats embeddings to align with the number of UNet cross-attention queries.

Routing Text Embeddings Through DiffusionWrapper

The DiffusionWrapper class in ldm/models/diffusion/ddpm.py serves as the interface between the conditioning encoder and the UNet. When conditioning_key is set to 'crossattn' (the default for text-to-image generation), the wrapper concatenates conditioning tensors and passes them as the context parameter.

class DiffusionWrapper(pl.LightningModule):
    def forward(self, x, t, c_concat: list = None, c_crossattn: list = None):
        if self.conditioning_key == 'crossattn':
            cc = torch.cat(c_crossattn, 1)          # → [B, K, 768]

            out = self.diffusion_model(x, t, context=cc)

The c_crossattn list contains the CLIP text embeddings produced by FrozenCLIPTextEmbedder.encode(). The concatenation allows multiple conditioning sources, though standard text-to-image uses a single text embedding tensor.

Cross-Attention Conditioning in the UNet

Inside the UNet architecture defined in ldm/modules/diffusionmodules/openaimodel.py, the UNetModel receives the context tensor and distributes it to spatial transformer blocks throughout the encoder, bottleneck, and decoder paths.

def forward(self, x, timesteps=None, context=None, y=None, **kwargs):
    h = self.input_blocks[0](x, emb)               # first block, no cross-attn

    for module in self.input_blocks[1:]:
        h = module(h, emb, context)                # text context injected here

    h = self.middle_block(h, emb, context)        # bottleneck attention

    for module in self.output_blocks:
        h = module(h, emb, context)                # decoder attention

The actual cross-attention computation occurs in SpatialTransformer layers (enabled when use_spatial_transformer=True), which utilize the CrossAttention class from ldm/modules/attention.py. This mechanism:

1. Computes queries from the latent feature maps (spatial dimensions).
2. Computes keys and values from the CLIP text embeddings (context).
3. Performs scaled dot-product attention between spatial features and text semantics.

This architecture allows every spatial location in the latent image to attend to relevant tokens in the text prompt, effectively steering the denoising process toward semantically appropriate outputs.

Complete Conditioning Pipeline

The end-to-end text conditioning flow follows this sequence:

1. Text Encoding: The prompt is processed by FrozenCLIPTextEmbedder.encode() to produce a [B, K, 768] tensor.
2. Wrapper Injection: DiffusionWrapper.forward() receives the embedding via c_crossattn and routes it to the UNet as context.
3. Cross-Attention: The UNet's residual blocks attend to the text embedding through SpatialTransformer layers while denoising the latent representation.
4. Generation: The conditioned UNet predicts noise residuals, guiding the DDIM/DDPM sampler toward a latent representation matching the text description.

Practical Implementation Example

Below is a minimal reproduction of the conditioning flow used in the txt2img.py inference script:

import torch
from ldm.util import instantiate_from_config
from ldm.modules.encoders.modules import FrozenCLIPTextEmbedder

# Initialize frozen CLIP ViT-L/14 encoder

text_encoder = FrozenCLIPTextEmbedder(version="ViT-L/14", device="cuda")
text_encoder.freeze()
text_encoder.eval()

# Encode prompt

prompt = ["a fantasy landscape with mountains, sunrise"]
text_emb = text_encoder.encode(prompt)  # → [1, 1, 768]

# Load diffusion model (DiffusionWrapper + UNet)

ckpt = torch.load("model.ckpt", map_location="cpu")
model = instantiate_from_config(ckpt["config"].model)
model.load_state_dict(ckpt["state_dict"], strict=False)
model.cuda().eval()

# Prepare random latent

latent_shape = (4, 64, 64)  # Channels, height/8, width/8 for 512px images

z = torch.randn(1, *latent_shape, device="cuda")

# Single denoising step with text conditioning

t = torch.tensor([999], device="cuda")  # Final timestep

with torch.no_grad():
    noise_pred = model(z, t, c_crossattn=[text_emb])

This pattern repeats across all timesteps in the sampling loop, with the text embedding providing consistent semantic guidance throughout the reverse diffusion process.

Summary

• CLIP ViT-L/14 encodes text prompts into 768-dimensional embeddings without gradient updates during diffusion training.
• FrozenCLIPTextEmbedder in ldm/modules/encoders/modules.py handles tokenization and produces [B, K, 768] conditioning tensors.
• DiffusionWrapper routes text embeddings to the UNet via the context parameter when conditioning_key='crossattn'.
• CrossAttention layers in ldm/modules/attention.py fuse spatial latent features with text semantics at every UNet block.

Frequently Asked Questions

What is the output dimension of CLIP ViT-L/14 text embeddings in Stable Diffusion?

The CLIP ViT-L/14 text encoder produces 768-dimensional embedding vectors. After processing through FrozenCLIPTextEmbedder.encode(), these are shaped as [batch_size, context_length, 768] (typically [1, 77, 768] when accounting for the full token sequence), providing rich semantic conditioning for the UNet.

Why is the CLIP text encoder kept frozen during training?

The FrozenCLIPTextEmbedder freezes CLIP weights to preserve the pre-trained alignment between text and visual concepts established during CLIP's contrastive pre-training. This prevents catastrophic forgetting of semantic relationships and stabilizes diffusion training, allowing the UNet to learn how to interpret fixed text embeddings rather than co-evolving the encoder.

How does cross-attention differ from self-attention in Stable Diffusion's UNet?

Self-attention layers in the UNet compute attention weights using queries, keys, and values all derived from the latent feature maps themselves, modeling spatial relationships within the image. Cross-attention layers (implemented in ldm/modules/attention.py) compute queries from latent features but derive keys and values from the CLIP text embeddings (context), effectively injecting external semantic information into the generation process.

Can Stable Diffusion use text encoders other than CLIP ViT-L/14?

While the CompVis/stable-diffusion repository specifically implements FrozenCLIPTextEmbedder for OpenAI's CLIP ViT-L/14, the modular architecture allows substitution of alternative encoders by implementing the same encode() interface returning [B, K, dim] tensors. Later Stable Diffusion variants utilize OpenCLIP models with different embedding dimensions, requiring corresponding adjustments to the UNet's cross-attention projection layers to match the new conditioning dimension.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}