How to Integrate Custom VQ-Perceptual and Contrastive Perceptual Losses in Stable Diffusion Training

You can integrate VQ-perceptual and contrastive perceptual losses into Stable Diffusion training by referencing the built-in loss classes in the YAML configuration file under model.params.losses, or by programmatically instantiating them in a custom LightningModule.

The CompVis/stable-diffusion repository uses the Lightning-Diffusion-Models (ldm) framework, which treats loss functions as pluggable modules configured through YAML files or direct Python instantiation. The codebase includes ready-made implementations for both VQ-perceptual and contrastive perceptual losses that integrate seamlessly with the standard training loop.

Understanding the Loss Architecture in Stable Diffusion

Stable Diffusion’s training pipeline constructs a loss dictionary (self.losses) inside the LightningModule during configure_optimizers. In ldm/models/diffusion/ddpm.py, the training step iterates over this dictionary, computes each loss component, applies a scalar weight, and sums the results into the final optimization target.

This architecture allows you to inject custom perceptual losses without modifying the core training logic. Both the VQ-perceptual and contrastive perceptual implementations inherit from torch.nn.Module and expose a standard forward method that accepts model predictions and ground-truth latent tensors.

Built-in Perceptual Loss Implementations

The repository provides two specialized perceptual loss classes in ldm/modules/losses/:

VQ-Perceptual Loss

Located in ldm/modules/losses/vqperceptual.py, the VQLoss class computes a perceptual distance using a VQ-GAN encoder. It compares high-level features between generated and target images to enforce semantic fidelity beyond pixel-level reconstruction.

Contrastive Perceptual Loss

Found in ldm/modules/losses/contperceptual.py, the ContPerceptualLoss class implements a contrastive learning approach that learns feature embeddings where matching image pairs are pulled closer together while non-matching pairs are pushed apart in latent space.

Method 1: Configuring Losses via YAML

The most common integration method uses the YAML configuration system. You can modify any training config in configs/latent-diffusion/ (such as txt2img-1p4B-finetune.yaml) to include perceptual losses under the model.params.losses section.

model:
  target: ldm.models.diffusion.ddpm.DDPM
  params:
    # ... existing model params ...

    losses:
      # Existing reconstruction loss (usually L2)

      reconstruction:
        target: torch.nn.MSELoss
        weight: 1.0

      # ---- VQ-Perceptual loss ----

      vq_perceptual:
        target: ldm.modules.losses.vqperceptual.VQLoss
        weight: 0.1                # adjust to balance with other terms

        params:
          perceptual_weight: 1.0   # internal weighting used inside VQLoss

      # ---- Contrastive Perceptual loss ----

      contrastive_perceptual:
        target: ldm.modules.losses.contperceptual.ContPerceptualLoss
        weight: 0.05               # tune this value experimentally

        params:
          temperature: 0.07
          projection_dim: 256

Key configuration parameters:

• target: The full Python import path to the loss class (e.g., ldm.modules.losses.vqperceptual.VQLoss).
• weight: The scalar multiplier applied to the loss before summing with other terms.
• params: Constructor arguments forwarded directly to the loss class __init__ method.

Method 2: Programmatic Integration in Python

For custom training scripts that bypass YAML configuration, instantiate the losses directly and assign them to the model:

from ldm.modules.losses.vqperceptual import VQLoss
from ldm.modules.losses.contperceptual import ContPerceptualLoss
import torch

# Inside your LightningModule setup

model.losses = {
    "reconstruction": torch.nn.MSELoss(),
    "vq_perceptual": VQLoss(perceptual_weight=1.0),
    "contrastive_perceptual": ContPerceptualLoss(temperature=0.07, projection_dim=256),
}

# Define scalar weights for aggregation

model.loss_weights = {
    "reconstruction": 1.0,
    "vq_perceptual": 0.1,
    "contrastive_perceptual": 0.05,
}

The training loop in ldm/models/diffusion/ddpm.py aggregates these losses using a pattern similar to:

def training_step(self, batch, batch_idx):
    x, y = batch
    pred = self(x)
    
    loss = 0.0
    for name, loss_fn in self.losses.items():
        cur = loss_fn(pred, y)
        loss += self.loss_weights[name] * cur
    return loss

Verification and Training Best Practices

After integrating custom perceptual losses, follow these steps to ensure stable training:

1. Run a sanity check with --max_epochs 1 and verify that separate loss entries for vq_perceptual and contrastive_perceptual appear in the logs.
2. Monitor GPU memory usage—perceptual losses require additional encoder networks that increase VRAM consumption. Enable gradient checkpointing if necessary.
3. Tune weights gradually—perceptual losses can dominate the optimization landscape if weighted too heavily. Start with small multipliers (0.01–0.1) and increase based on validation FID scores or visual quality metrics.

Summary

• VQ-perceptual loss (ldm/modules/losses/vqperceptual.py) and contrastive perceptual loss (ldm/modules/losses/contperceptual.py) are built into the Stable Diffusion codebase and ready for immediate use.
• Add losses via YAML configuration under model.params.losses using the target, weight, and params keys.
• Alternatively, instantiate losses programmatically and populate self.losses and self.loss_weights in your LightningModule.
• The training loop in ldm/models/diffusion/ddpm.py automatically aggregates weighted loss terms without requiring modifications to the core logic.
• Begin with conservative weight values (0.05–0.1) to prevent perceptual terms from overwhelming the reconstruction objective.

Frequently Asked Questions

Where are the perceptual loss classes defined in the Stable Diffusion codebase?

The VQ-perceptual loss is implemented in ldm/modules/losses/vqperceptual.py and the contrastive perceptual loss is in ldm/modules/losses/contperceptual.py. Both files define torch.nn.Module subclasses that integrate with the Lightning-Diffusion-Models framework used throughout the CompVis/stable-diffusion repository.

How do I balance the weight of perceptual losses against reconstruction loss?

Set the weight parameter in the YAML configuration (or the value in loss_weights for programmatic setup) to a value between 0.01 and 0.1 initially. Monitor validation metrics such as FID or visual coherence, then adjust upward if the model produces blurry outputs or downward if training becomes unstable.

Can I use multiple perceptual losses simultaneously during training?

Yes, the loss dictionary architecture supports multiple concurrent losses. Simply add separate entries for vq_perceptual and contrastive_perceptual (or custom implementations) in the same losses configuration block. The training step will automatically sum all weighted contributions into the final loss scalar.

What is the difference between VQ-perceptual and contrastive perceptual loss?

VQ-perceptual loss uses a VQ-GAN encoder to compute feature-space distances between generated and target images, emphasizing high-level visual similarity. Contrastive perceptual loss instead learns discriminative embeddings that minimize distance between matching pairs while maximizing distance between non-matching pairs, which can improve semantic alignment in the latent space.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}