# How to Integrate Custom VQ-Perceptual and Contrastive Perceptual Losses in Stable Diffusion Training

> Learn how to integrate custom VQ-perceptual and contrastive perceptual losses into Stable Diffusion training using YAML configurations or programmatic instantiation in a custom LightningModule.

- Repository: [CompVis - Computer Vision and Learning LMU Munich/stable-diffusion](https://github.com/CompVis/stable-diffusion)
- Tags: how-to-guide
- Published: 2026-03-02

---

**You can integrate VQ-perceptual and contrastive perceptual losses into Stable Diffusion training by referencing the built-in loss classes in the YAML configuration file under `model.params.losses`, or by programmatically instantiating them in a custom LightningModule.**

The CompVis/stable-diffusion repository uses the **Lightning-Diffusion-Models (ldm)** framework, which treats loss functions as pluggable modules configured through YAML files or direct Python instantiation. The codebase includes ready-made implementations for both VQ-perceptual and contrastive perceptual losses that integrate seamlessly with the standard training loop.

## Understanding the Loss Architecture in Stable Diffusion

Stable Diffusion’s training pipeline constructs a loss dictionary (`self.losses`) inside the LightningModule during `configure_optimizers`. In [`ldm/models/diffusion/ddpm.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/models/diffusion/ddpm.py), the training step iterates over this dictionary, computes each loss component, applies a scalar weight, and sums the results into the final optimization target.

This architecture allows you to inject custom perceptual losses without modifying the core training logic. Both the VQ-perceptual and contrastive perceptual implementations inherit from `torch.nn.Module` and expose a standard `forward` method that accepts model predictions and ground-truth latent tensors.

## Built-in Perceptual Loss Implementations

The repository provides two specialized perceptual loss classes in `ldm/modules/losses/`:

### VQ-Perceptual Loss

Located in [`ldm/modules/losses/vqperceptual.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/modules/losses/vqperceptual.py), the `VQLoss` class computes a perceptual distance using a VQ-GAN encoder. It compares high-level features between generated and target images to enforce semantic fidelity beyond pixel-level reconstruction.

### Contrastive Perceptual Loss

Found in [`ldm/modules/losses/contperceptual.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/modules/losses/contperceptual.py), the `ContPerceptualLoss` class implements a contrastive learning approach that learns feature embeddings where matching image pairs are pulled closer together while non-matching pairs are pushed apart in latent space.

## Method 1: Configuring Losses via YAML

The most common integration method uses the YAML configuration system. You can modify any training config in `configs/latent-diffusion/` (such as [`txt2img-1p4B-finetune.yaml`](https://github.com/CompVis/stable-diffusion/blob/main/txt2img-1p4B-finetune.yaml)) to include perceptual losses under the `model.params.losses` section.

```yaml
model:
  target: ldm.models.diffusion.ddpm.DDPM
  params:
    # ... existing model params ...

    losses:
      # Existing reconstruction loss (usually L2)

      reconstruction:
        target: torch.nn.MSELoss
        weight: 1.0

      # ---- VQ-Perceptual loss ----

      vq_perceptual:
        target: ldm.modules.losses.vqperceptual.VQLoss
        weight: 0.1                # adjust to balance with other terms

        params:
          perceptual_weight: 1.0   # internal weighting used inside VQLoss

      # ---- Contrastive Perceptual loss ----

      contrastive_perceptual:
        target: ldm.modules.losses.contperceptual.ContPerceptualLoss
        weight: 0.05               # tune this value experimentally

        params:
          temperature: 0.07
          projection_dim: 256

```

**Key configuration parameters:**

- **`target`**: The full Python import path to the loss class (e.g., `ldm.modules.losses.vqperceptual.VQLoss`).
- **`weight`**: The scalar multiplier applied to the loss before summing with other terms.
- **`params`**: Constructor arguments forwarded directly to the loss class `__init__` method.

## Method 2: Programmatic Integration in Python

For custom training scripts that bypass YAML configuration, instantiate the losses directly and assign them to the model:

```python
from ldm.modules.losses.vqperceptual import VQLoss
from ldm.modules.losses.contperceptual import ContPerceptualLoss
import torch

# Inside your LightningModule setup

model.losses = {
    "reconstruction": torch.nn.MSELoss(),
    "vq_perceptual": VQLoss(perceptual_weight=1.0),
    "contrastive_perceptual": ContPerceptualLoss(temperature=0.07, projection_dim=256),
}

# Define scalar weights for aggregation

model.loss_weights = {
    "reconstruction": 1.0,
    "vq_perceptual": 0.1,
    "contrastive_perceptual": 0.05,
}

```

The training loop in [`ldm/models/diffusion/ddpm.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/models/diffusion/ddpm.py) aggregates these losses using a pattern similar to:

```python
def training_step(self, batch, batch_idx):
    x, y = batch
    pred = self(x)
    
    loss = 0.0
    for name, loss_fn in self.losses.items():
        cur = loss_fn(pred, y)
        loss += self.loss_weights[name] * cur
    return loss

```

## Verification and Training Best Practices

After integrating custom perceptual losses, follow these steps to ensure stable training:

1. **Run a sanity check** with `--max_epochs 1` and verify that separate loss entries for `vq_perceptual` and `contrastive_perceptual` appear in the logs.
2. **Monitor GPU memory usage**—perceptual losses require additional encoder networks that increase VRAM consumption. Enable gradient checkpointing if necessary.
3. **Tune weights gradually**—perceptual losses can dominate the optimization landscape if weighted too heavily. Start with small multipliers (0.01–0.1) and increase based on validation FID scores or visual quality metrics.

## Summary

- **VQ-perceptual loss** ([`ldm/modules/losses/vqperceptual.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/modules/losses/vqperceptual.py)) and **contrastive perceptual loss** ([`ldm/modules/losses/contperceptual.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/modules/losses/contperceptual.py)) are built into the Stable Diffusion codebase and ready for immediate use.
- Add losses via YAML configuration under `model.params.losses` using the `target`, `weight`, and `params` keys.
- Alternatively, instantiate losses programmatically and populate `self.losses` and `self.loss_weights` in your LightningModule.
- The training loop in [`ldm/models/diffusion/ddpm.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/models/diffusion/ddpm.py) automatically aggregates weighted loss terms without requiring modifications to the core logic.
- Begin with conservative weight values (0.05–0.1) to prevent perceptual terms from overwhelming the reconstruction objective.

## Frequently Asked Questions

### Where are the perceptual loss classes defined in the Stable Diffusion codebase?

The VQ-perceptual loss is implemented in [`ldm/modules/losses/vqperceptual.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/modules/losses/vqperceptual.py) and the contrastive perceptual loss is in [`ldm/modules/losses/contperceptual.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/modules/losses/contperceptual.py). Both files define `torch.nn.Module` subclasses that integrate with the Lightning-Diffusion-Models framework used throughout the CompVis/stable-diffusion repository.

### How do I balance the weight of perceptual losses against reconstruction loss?

Set the `weight` parameter in the YAML configuration (or the value in `loss_weights` for programmatic setup) to a value between 0.01 and 0.1 initially. Monitor validation metrics such as FID or visual coherence, then adjust upward if the model produces blurry outputs or downward if training becomes unstable.

### Can I use multiple perceptual losses simultaneously during training?

Yes, the loss dictionary architecture supports multiple concurrent losses. Simply add separate entries for `vq_perceptual` and `contrastive_perceptual` (or custom implementations) in the same `losses` configuration block. The training step will automatically sum all weighted contributions into the final loss scalar.

### What is the difference between VQ-perceptual and contrastive perceptual loss?

VQ-perceptual loss uses a VQ-GAN encoder to compute feature-space distances between generated and target images, emphasizing high-level visual similarity. Contrastive perceptual loss instead learns discriminative embeddings that minimize distance between matching pairs while maximizing distance between non-matching pairs, which can improve semantic alignment in the latent space.