# How to Debug Stable Diffusion Mode Collapse, Artifacts, and Poor Prompt Adherence

> Debug Stable Diffusion mode collapse artifacts and prompt adherence issues. Tune guidance scale, increase diffusion steps, and verify text conditioning for better results.

- Repository: [CompVis - Computer Vision and Learning LMU Munich/stable-diffusion](https://github.com/CompVis/stable-diffusion)
- Tags: how-to-guide
- Published: 2026-03-02

---

**Tune the classifier-free guidance scale between 5.0 and 10.0, increase diffusion steps to 150–200 for complex prompts, and verify that text conditioning tensors maintain shape (1, 77, 768) to eliminate repetitive patterns and alignment drift.**

Debugging generation failures in the CompVis/stable-diffusion repository requires understanding how the UNet fuses latent tensors with CLIP text embeddings during iterative denoising. Mode collapse and prompt adherence failures typically stem from misconfigured guidance scales, truncated conditioning sequences, or insufficient sampling steps in the PLMS or DDIM samplers.

## Understanding Mode Collapse and Artifact Sources

Mode collapse manifests as repetitive patterns, blocky textures, or color saturation explosions. These artifacts originate in the interaction between the classifier-free guidance (CFG) mechanism and the stochastic sampling trajectory.

### High Guidance Scale Amplification

When the guidance scale exceeds the training distribution’s comfort zone—typically above 10.0—the subtraction term in the CFG formula dominates the gradient updates. In [`ldm/models/diffusion/plms.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/models/diffusion/plms.py) (lines 78‑84), the sampler concatenates conditional and unconditional predictions, then mixes them using the scalar `unconditional_guidance_scale`. Excessive values amplify high-frequency errors the UNet never observed during training, pushing latents into repetitive basins. If you observe blocky repeats, reduce `--scale` to 5.0–6.0 in [`scripts/txt2img.py`](https://github.com/CompVis/stable-diffusion/blob/main/scripts/txt2img.py) (line 203).

### Step Count and Stochasticity

Too few diffusion steps or an overly large **eta** (DDIM stochasticity parameter) truncates the denoising trajectory before the latent aligns with the conditioning signal. The `make_schedule` method in the sampler configures `ddim_num_steps` and `ddim_eta` (referenced around lines 24‑28). For production prompts, escalate from the default 50 steps to 150 or 200 to allow full convergence.

### Weight Initialization Integrity

Artifacts also surface when loading checkpoints with mismatched UNet architectures. The `UNetModel` class defined in [`ldm/modules/diffusionmodules/openaimodel.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/modules/diffusionmodules/openaimodel.py) (lines 13‑16) expects specific `num_heads` and transformer depth. A mismatch corrupts the residual pathways that fuse text and image features, producing incoherent activations.

## Diagnosing Poor Prompt Adherence

When outputs drift from the textual description—ignoring style tokens or omitting objects—the conditioning pathway is failing to override the unconditional prior.

### Conditioning Strength and CFG Scale

Conversely to mode collapse, guidance scales below 5.0 allow the unconditional noise prediction to dominate, effectively disregarding the prompt. The CLI exposes this via `--scale` in [`scripts/txt2img.py`](https://github.com/CompVis/stable-diffusion/blob/main/scripts/txt2img.py). Raise values to 8.0–10.0 to enforce strict adherence, though monitor for the repetitive artifacts described above.

### Text Encoder Token Limits

The CLIP text encoder (`FrozenCLIPTextEmbedder` in [`ldm/modules/encoders/modules.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/modules/encoders/modules.py)) truncates inputs at 77 tokens. Exceeding this limit cuts off trailing prompt keywords, causing partial adherence. Verify your conditioning tensor shape after calling `model.get_learned_conditioning` in [`ldm/models/diffusion/ddpm.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/models/diffusion/ddpm.py) (line 551); the output must be `(1, 77, 768)` for CLIP‑ViT‑L/14. Deviation indicates truncation.

### Sampling Trajectory Length

Inadequate step counts prevent the latent from fully aligning with the conditioning signal. The `get_learned_conditioning` output flows into the UNet’s cross-attention layers; if the sampler terminates too early (low `S` in `PLMSSampler`), the cross-attention scores never saturate to match the text embedding.

## Critical Code Paths for Debugging

Understanding these three components isolates whether the issue lies in guidance math, text encoding, or network architecture:

- **UNetModel** ([`ldm/modules/diffusionmodules/openaimodel.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/modules/diffusionmodules/openaimodel.py), lines 13‑16): The core denoising network. Its residual blocks and attention layers determine how effectively noisy latents fuse with textual embeddings.
- **Classifier-Free Guidance** ([`ldm/models/diffusion/plms.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/models/diffusion/plms.py), lines 78‑84): Implements the `unconditional_guidance_scale` mixing logic. This is where excessive scale values perturb the gradient trajectory.
- **Conditioning Extraction** ([`ldm/models/diffusion/ddpm.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/models/diffusion/ddpm.py), line 551): The `get_learned_conditioning` method bridges text prompts to the UNet. Failures here propagate as prompt drift.

## Practical Debugging Checklist

Follow this sequence to isolate mode collapse versus adherence failures:

1. **Baseline CFG at 7.5**: Start with the default guidance scale in [`scripts/txt2img.py`](https://github.com/CompVis/stable-diffusion/blob/main/scripts/txt2img.py). Adjust downward to reduce blocky artifacts, upward to correct prompt drift.
2. **Increase Diffusion Steps**: Move from 50 to 150–200 steps for complex scenes with multiple entities.
3. **Validate Tokenization**: Check that prompts do not exceed 77 tokens; inspect `cond.shape` after `get_learned_conditioning` to confirm `(1, 77, 768)`.
4. **Checkpoint Compatibility**: Ensure the loaded `.ckpt` matches the config’s UNet depth and attention resolutions to avoid weight loading errors.
5. **Lock Randomness**: Set `--seed` and `--ddim_eta 0` to eliminate stochasticity as a variable during reproduction.

## Implementation Fixes and Code Examples

### Command-Line Debugging with Tuned Parameters

Adjust scale and steps in the standard txt2img invocation:

```bash
python scripts/txt2img.py \
  --prompt "A serene sunrise over misty mountains, cinematic lighting" \
  --ckpt_path models/ldm/stable-diffusion-v1/model.ckpt \
  --outdir outputs/debug \
  --H 512 --W 512 \
  --ddim_steps 150 \
  --scale 8.0 \
  --seed 42

```

Key parameters: `--scale` controls CFG strength, `--ddim_steps` sets trajectory length, `--seed` ensures determinism.

### Programmatic Sampling with Custom Guidance

For granular control, instantiate the `PLMSSampler` directly and inspect conditioning shapes:

```python
import torch
from ldm.models.diffusion.plms import PLMSSampler
from ldm.util import instantiate_from_config
from omegaconf import OmegaConf

config = OmegaConf.load("configs/stable-diffusion/v1-inference.yaml")
model = instantiate_from_config(config.model)
sd = torch.load("models/ldm/stable-diffusion-v1/model.ckpt")["state_dict"]
model.load_state_dict(sd, strict=False)
model.eval().cuda()

sampler = PLMSSampler(model)

prompt = "A futuristic cityscape at night, neon lights"
cond = model.get_learned_conditioning([prompt]).cuda()
uc = model.get_learned_conditioning([""]).repeat(cond.shape[0], 1, 1).cuda()

# Verify shape before sampling

assert cond.shape == (1, 77, 768), "Prompt truncation detected"

samples, _ = sampler.sample(
    S=200,
    batch_size=1,
    shape=(4, 64, 64),
    conditioning=cond,
    unconditional_conditioning=uc,
    unconditional_guidance_scale=9.0,
    verbose=False,
)

x_samples_ddim = model.decode_first_stage(samples)
x_img = torch.clamp((x_samples_ddim + 1.0) / 2.0, 0, 1)

```

### Systematic CFG Sweep for Optimal Balance

Iterate through scales to find the lowest value that maintains prompt fidelity without triggering repetition:

```python
for scale in [5.0, 6.0, 7.0, 8.0]:
    samples, _ = sampler.sample(
        S=100,
        batch_size=1,
        shape=(4, 64, 64),
        conditioning=cond,
        unconditional_conditioning=uc,
        unconditional_guidance_scale=scale,
    )
    # Decode and save outputs for visual comparison

```

## Summary

- **Classifier-Free Guidance** acts as a fidelity lever: values below 5.0 cause prompt drift, while values above 10.0 induce mode collapse and blocky artifacts.
- **Diffusion step count** directly impacts convergence; complex prompts require 150–200 steps rather than the default 50 to avoid truncated trajectories.
- **Text conditioning shape** must be `(1, 77, 768)`; truncation beyond 77 tokens severs prompt keywords from the conditioning signal.
- **Source files** [`ldm/models/diffusion/plms.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/models/diffusion/plms.py) (CFG logic), [`ldm/models/diffusion/ddpm.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/models/diffusion/ddpm.py) (embedding extraction), and [`ldm/modules/diffusionmodules/openaimodel.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/modules/diffusionmodules/openaimodel.py) (UNet architecture) contain the critical parameters governing generation quality.

## Frequently Asked Questions

### What causes mode collapse in Stable Diffusion?

Mode collapse occurs when excessively high guidance scales (>10.0) amplify the unconditional gradient subtraction term in `PLMSSampler`, driving the denoising trajectory into repetitive latent basins the UNet never encountered during training. Reduce `--scale` to 5.0–6.0 or increase diffusion steps to restore diversity.

### Why does my output ignore parts of the prompt?

Poor adherence indicates weak conditioning, typically from guidance scales below 5.0 that allow the unconditional prior to dominate, or from prompt truncation exceeding the 77-token limit of the CLIP text encoder in [`ldm/modules/encoders/modules.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/modules/encoders/modules.py). Verify `cond.shape == (1, 77, 768)` after calling `get_learned_conditioning` in [`ldm/models/diffusion/ddpm.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/models/diffusion/ddpm.py).

### How many diffusion steps are required to avoid artifacts?

While 50 steps suffice for simple compositions, intricate scenes with multiple entities require 150–200 steps to allow the latent to fully align with text embeddings. Fewer steps truncate the schedule in `make_schedule`, preventing the cross-attention layers from saturating against the conditioning signal.

### What is the relationship between eta and generation stability?

The **eta** parameter controls stochasticity in DDIM-style samplers; values near 1.0 inject noise that can destabilize the trajectory, while `--ddim_eta 0` enforces deterministic sampling. High eta combined with low step counts exacerbates both mode collapse and prompt drift by limiting latent space exploration.