How to Perform Inpainting with Custom Masks Using Stable Diffusion

Stable Diffusion performs inpainting by conditioning a latent diffusion model on a masked image and binary mask, concatenating the encoded image latent with a downsampled mask to create a 7-channel input, and using DDIMSampler to generate content for the masked regions.

The CompVis/stable-diffusion repository provides a complete implementation for performing inpainting with custom masks using Stable Diffusion. This latent diffusion approach allows you to replace arbitrary regions of an image with AI-generated content by providing a binary mask that defines the inpainting area.

Understanding the Inpainting Architecture

The inpainting implementation in Stable Diffusion extends the base latent diffusion model to condition on both the image and a mask. According to the source code in scripts/inpaint.py, the model processes a masked image (the original image with the masked region zeroed out) alongside a binary mask indicating which pixels need regeneration.

The core architecture relies on the concat_mode: true configuration found in models/ldm/inpainting_big/config.yaml (lines 14-15). This mode enables the model to accept a 7-channel input tensor composed of:

• 3 channels from the encoded image latent
• 1 channel from the downsampled binary mask
• 3 channels from the masked image encoding

Data Preparation with make_batch

Before inference, you must prepare the image and mask using the make_batch function defined in scripts/inpaint.py (lines 11-30). This function handles normalization and tensor creation.

The function performs the following operations:

• Loads the RGB image and grayscale mask
• Normalizes both to the [0, 1] range
• Creates a masked image by multiplying the original image with (1 - mask)
• Scales tensors to the model's expected [-1, 1] range

Building the Conditioning with concat_mode

The conditioning logic, implemented in scripts/inpaint.py (lines 75-82), constructs the input for the diffusion model. The process involves:

1. Encoding the masked image using the cond_stage_model (a VQ-autoencoder)
2. Downsampling the binary mask to match the latent spatial dimensions using torch.nn.functional.interpolate
3. Concatenating the encoded latent (3 channels) with the downsampled mask (1 channel) to form a 4-channel conditioning tensor

When concat_mode: true is set in the configuration, the model automatically handles the concatenation to create the final 7-channel input during the diffusion process.

Sampling with DDIMSampler

The inpainting process uses DDIMSampler from ldm/models/diffusion/ddim.py to generate latent samples. As shown in scripts/inpaint.py (lines 66-86), the sampler draws latent representations conditioned on the concatenated feature map.

After sampling, the pipeline:

1. Decodes the latent sample back to pixel space using the model's first-stage decoder
2. Blends the generated content with the original unmasked region using the binary mask
3. Outputs the final inpainted image

Complete Implementation: Command Line and Python API

You can perform inpainting using either the command-line interface or a Python API.

Command-Line Usage

The simplest way to run inpainting is using the provided script:

python scripts/inpaint.py \
    --indir /path/to/pairs \
    --outdir /path/to/results \
    --steps 50

The input directory must contain image-mask pairs with matching filenames (e.g., example.png and example_mask.png). The script automatically loads the inpainting model configuration from models/ldm/inpainting_big/config.yaml and the checkpoint from models/ldm/inpainting_big/last.ckpt using instantiate_from_config from ldm/util.py.

Python API Implementation

For programmatic control, use the model components directly:

import torch
from pathlib import Path
from ldm.util import instantiate_from_config
from omegaconf import OmegaConf
from scripts.inpaint import make_batch
from ldm.models.diffusion.ddim import DDIMSampler

# 1️⃣ Load model

cfg = OmegaConf.load("models/ldm/inpainting_big/config.yaml")
model = instantiate_from_config(cfg.model)
ckpt = torch.load("models/ldm/inpainting_big/last.ckpt")["state_dict"]
model.load_state_dict(ckpt, strict=False)
model = model.to("cuda" if torch.cuda.is_available() else "cpu")
sampler = DDIMSampler(model)

# 2️⃣ Prepare inputs

image_path = Path("my_image.png")
mask_path  = Path("my_mask.png")          # grayscale, 0–255 (≥128 = masked)

batch = make_batch(str(image_path), str(mask_path),
                   device=model.device)

# 3️⃣ Build conditioning (exactly as the script does)

c = model.cond_stage_model.encode(batch["masked_image"])
c_mask = torch.nn.functional.interpolate(batch["mask"],
                                         size=c.shape[-2:])
c = torch.cat([c, c_mask], dim=1)        # 7-channel conditioning

# 4️⃣ Sample inpainted latent

shape = (c.shape[1]-1,) + c.shape[2:]    # latent channels = 4

samples, _ = sampler.sample(S=50,            # number of DDIM steps

                            conditioning=c,
                            batch_size=c.shape[0],
                            shape=shape,
                            verbose=False)

# 5️⃣ Decode and blend

x_latent = model.decode_first_stage(samples)
pred = torch.clamp((x_latent + 1) / 2, 0, 1)
orig = torch.clamp((batch["image"] + 1) / 2, 0, 1)
mask = torch.clamp((batch["mask"] + 1) / 2, 0, 1)
inpainted = (1 - mask) * orig + mask * pred
inpainted = (inpainted.cpu().numpy()[0] * 255).astype("uint8")
Image.fromarray(inpainted).save("result.png")

Creating Custom Masks Programmatically

You can generate custom masks using PIL and NumPy:

import numpy as np
from PIL import Image

# Load the original image to get size

w, h = Image.open("my_image.png").size

# Create a binary mask (e.g., a circular hole)

mask = np.zeros((h, w), dtype=np.uint8)
cx, cy, r = w // 2, h // 2, min(w, h) // 4
y, x = np.ogrid[:h, :w]
mask[(x - cx) ** 2 + (y - cy) ** 2 <= r ** 2] = 255
Image.fromarray(mask).save("my_mask.png")

The generated my_mask.png can now be paired with the image and fed to the inpainting script.

Summary

• Stable Diffusion inpainting uses a latent diffusion model that conditions on a masked image and binary mask, implemented in scripts/inpaint.py.
• Data preparation requires normalizing images to [-1, 1] and creating masked images via the make_batch function.
• Conditioning relies on concat_mode: true in the model configuration to concatenate the encoded image latent with the downsampled mask, creating a 7-channel input.
• Sampling uses DDIMSampler from ldm/models/diffusion/ddim.py to generate latent representations, which are then decoded and blended with the original image using the mask.
• You can run inpainting via the command-line interface or a Python API, and generate custom masks programmatically using NumPy and PIL.

Frequently Asked Questions

What file format should I use for custom masks?

Custom masks should be saved as grayscale PNG images where pixel values range from 0 to 255. The make_batch function in scripts/inpaint.py interprets values greater than or equal to 128 as masked regions (value 1.0) and values below 128 as unmasked regions (value 0.0). Ensure your mask dimensions match the input image dimensions exactly.

How does the concat_mode configuration affect the model?

The concat_mode: true setting in models/ldm/inpainting_big/config.yaml enables the model to process a 7-channel input tensor instead of the standard 4-channel latent. According to the conditioning code in scripts/inpaint.py (lines 75-82), this concatenates the encoded masked image (3 channels), the downsampled binary mask (1 channel), and the latent representation (3 channels), allowing the U-Net to attend to both the masked content and the mask geometry simultaneously.

Can I use a different sampler instead of DDIMSampler?

Yes, while the reference implementation in scripts/inpaint.py uses DDIMSampler from ldm/models/diffusion/ddim.py, you can substitute other samplers from the ldm/models/diffusion/ module such as PLMSSampler or DDPMSampler. However, you must ensure the sampler supports conditioning on the concatenated 7-channel tensor structure required by the inpainting model configuration. The DDIM sampler is preferred for inpainting due to its deterministic nature and fewer sampling steps required for high-quality results.

What is the expected resolution for input images and masks?

The inpainting model typically processes inputs at 512×512 pixels resolution, though the make_batch function will automatically resize inputs if needed. For optimal results, provide images and masks at 512×512 resolution. The mask is downsampled via torch.nn.functional.interpolate to match the latent spatial dimensions (typically 64×64 for 512×512 inputs with an 8× downsampling factor) before being concatenated with the image encoding.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}