# How to Perform Inpainting with Custom Masks Using Stable Diffusion

> Learn how to perform inpainting with custom masks using Stable Diffusion. This guide explains the process of conditioning the latent diffusion model effectively for precise image generation over masked areas. Get started now.

- Repository: [CompVis - Computer Vision and Learning LMU Munich/stable-diffusion](https://github.com/CompVis/stable-diffusion)
- Tags: how-to-guide
- Published: 2026-03-01

---

**Stable Diffusion performs inpainting by conditioning a latent diffusion model on a masked image and binary mask, concatenating the encoded image latent with a downsampled mask to create a 7-channel input, and using DDIMSampler to generate content for the masked regions.**

The CompVis/stable-diffusion repository provides a complete implementation for performing inpainting with custom masks using Stable Diffusion. This latent diffusion approach allows you to replace arbitrary regions of an image with AI-generated content by providing a binary mask that defines the inpainting area.

## Understanding the Inpainting Architecture

The inpainting implementation in Stable Diffusion extends the base latent diffusion model to condition on both the image and a mask. According to the source code in [`scripts/inpaint.py`](https://github.com/CompVis/stable-diffusion/blob/main/scripts/inpaint.py), the model processes a **masked image** (the original image with the masked region zeroed out) alongside a **binary mask** indicating which pixels need regeneration.

The core architecture relies on the `concat_mode: true` configuration found in [`models/ldm/inpainting_big/config.yaml`](https://github.com/CompVis/stable-diffusion/blob/main/models/ldm/inpainting_big/config.yaml) (lines 14-15). This mode enables the model to accept a 7-channel input tensor composed of:

- 3 channels from the encoded image latent
- 1 channel from the downsampled binary mask
- 3 channels from the masked image encoding

## Data Preparation with `make_batch`

Before inference, you must prepare the image and mask using the `make_batch` function defined in [`scripts/inpaint.py`](https://github.com/CompVis/stable-diffusion/blob/main/scripts/inpaint.py) (lines 11-30). This function handles normalization and tensor creation.

The function performs the following operations:

- Loads the RGB image and grayscale mask
- Normalizes both to the `[0, 1]` range
- Creates a masked image by multiplying the original image with `(1 - mask)`
- Scales tensors to the model's expected `[-1, 1]` range

## Building the Conditioning with `concat_mode`

The conditioning logic, implemented in [`scripts/inpaint.py`](https://github.com/CompVis/stable-diffusion/blob/main/scripts/inpaint.py) (lines 75-82), constructs the input for the diffusion model. The process involves:

1. Encoding the masked image using the `cond_stage_model` (a VQ-autoencoder)
2. Downsampling the binary mask to match the latent spatial dimensions using `torch.nn.functional.interpolate`
3. Concatenating the encoded latent (3 channels) with the downsampled mask (1 channel) to form a 4-channel conditioning tensor

When `concat_mode: true` is set in the configuration, the model automatically handles the concatenation to create the final 7-channel input during the diffusion process.

## Sampling with DDIMSampler

The inpainting process uses `DDIMSampler` from [`ldm/models/diffusion/ddim.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/models/diffusion/ddim.py) to generate latent samples. As shown in [`scripts/inpaint.py`](https://github.com/CompVis/stable-diffusion/blob/main/scripts/inpaint.py) (lines 66-86), the sampler draws latent representations conditioned on the concatenated feature map.

After sampling, the pipeline:

1. Decodes the latent sample back to pixel space using the model's first-stage decoder
2. Blends the generated content with the original unmasked region using the binary mask
3. Outputs the final inpainted image

## Complete Implementation: Command Line and Python API

You can perform inpainting using either the command-line interface or a Python API.

### Command-Line Usage

The simplest way to run inpainting is using the provided script:

```bash
python scripts/inpaint.py \
    --indir /path/to/pairs \
    --outdir /path/to/results \
    --steps 50

```

The input directory must contain image-mask pairs with matching filenames (e.g., `example.png` and `example_mask.png`). The script automatically loads the inpainting model configuration from [`models/ldm/inpainting_big/config.yaml`](https://github.com/CompVis/stable-diffusion/blob/main/models/ldm/inpainting_big/config.yaml) and the checkpoint from `models/ldm/inpainting_big/last.ckpt` using `instantiate_from_config` from [`ldm/util.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/util.py).

### Python API Implementation

For programmatic control, use the model components directly:

```python
import torch
from pathlib import Path
from ldm.util import instantiate_from_config
from omegaconf import OmegaConf
from scripts.inpaint import make_batch
from ldm.models.diffusion.ddim import DDIMSampler

# 1️⃣ Load model

cfg = OmegaConf.load("models/ldm/inpainting_big/config.yaml")
model = instantiate_from_config(cfg.model)
ckpt = torch.load("models/ldm/inpainting_big/last.ckpt")["state_dict"]
model.load_state_dict(ckpt, strict=False)
model = model.to("cuda" if torch.cuda.is_available() else "cpu")
sampler = DDIMSampler(model)

# 2️⃣ Prepare inputs

image_path = Path("my_image.png")
mask_path  = Path("my_mask.png")          # grayscale, 0–255 (≥128 = masked)

batch = make_batch(str(image_path), str(mask_path),
                   device=model.device)

# 3️⃣ Build conditioning (exactly as the script does)

c = model.cond_stage_model.encode(batch["masked_image"])
c_mask = torch.nn.functional.interpolate(batch["mask"],
                                         size=c.shape[-2:])
c = torch.cat([c, c_mask], dim=1)        # 7-channel conditioning

# 4️⃣ Sample inpainted latent

shape = (c.shape[1]-1,) + c.shape[2:]    # latent channels = 4

samples, _ = sampler.sample(S=50,            # number of DDIM steps

                            conditioning=c,
                            batch_size=c.shape[0],
                            shape=shape,
                            verbose=False)

# 5️⃣ Decode and blend

x_latent = model.decode_first_stage(samples)
pred = torch.clamp((x_latent + 1) / 2, 0, 1)
orig = torch.clamp((batch["image"] + 1) / 2, 0, 1)
mask = torch.clamp((batch["mask"] + 1) / 2, 0, 1)
inpainted = (1 - mask) * orig + mask * pred
inpainted = (inpainted.cpu().numpy()[0] * 255).astype("uint8")
Image.fromarray(inpainted).save("result.png")

```

### Creating Custom Masks Programmatically

You can generate custom masks using PIL and NumPy:

```python
import numpy as np
from PIL import Image

# Load the original image to get size

w, h = Image.open("my_image.png").size

# Create a binary mask (e.g., a circular hole)

mask = np.zeros((h, w), dtype=np.uint8)
cx, cy, r = w // 2, h // 2, min(w, h) // 4
y, x = np.ogrid[:h, :w]
mask[(x - cx) ** 2 + (y - cy) ** 2 <= r ** 2] = 255
Image.fromarray(mask).save("my_mask.png")

```

The generated `my_mask.png` can now be paired with the image and fed to the inpainting script.

## Summary

- **Stable Diffusion inpainting** uses a latent diffusion model that conditions on a masked image and binary mask, implemented in [`scripts/inpaint.py`](https://github.com/CompVis/stable-diffusion/blob/main/scripts/inpaint.py).
- **Data preparation** requires normalizing images to `[-1, 1]` and creating masked images via the `make_batch` function.
- **Conditioning** relies on `concat_mode: true` in the model configuration to concatenate the encoded image latent with the downsampled mask, creating a 7-channel input.
- **Sampling** uses `DDIMSampler` from [`ldm/models/diffusion/ddim.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/models/diffusion/ddim.py) to generate latent representations, which are then decoded and blended with the original image using the mask.
- You can run inpainting via the **command-line interface** or a **Python API**, and generate **custom masks** programmatically using NumPy and PIL.

## Frequently Asked Questions

### What file format should I use for custom masks?

Custom masks should be saved as **grayscale PNG images** where pixel values range from 0 to 255. The `make_batch` function in [`scripts/inpaint.py`](https://github.com/CompVis/stable-diffusion/blob/main/scripts/inpaint.py) interprets values greater than or equal to 128 as masked regions (value 1.0) and values below 128 as unmasked regions (value 0.0). Ensure your mask dimensions match the input image dimensions exactly.

### How does the `concat_mode` configuration affect the model?

The `concat_mode: true` setting in [`models/ldm/inpainting_big/config.yaml`](https://github.com/CompVis/stable-diffusion/blob/main/models/ldm/inpainting_big/config.yaml) enables the model to process a 7-channel input tensor instead of the standard 4-channel latent. According to the conditioning code in [`scripts/inpaint.py`](https://github.com/CompVis/stable-diffusion/blob/main/scripts/inpaint.py) (lines 75-82), this concatenates the encoded masked image (3 channels), the downsampled binary mask (1 channel), and the latent representation (3 channels), allowing the U-Net to attend to both the masked content and the mask geometry simultaneously.

### Can I use a different sampler instead of DDIMSampler?

Yes, while the reference implementation in [`scripts/inpaint.py`](https://github.com/CompVis/stable-diffusion/blob/main/scripts/inpaint.py) uses `DDIMSampler` from [`ldm/models/diffusion/ddim.py`](https://github.com/CompVis/stable-diffusion/blob/main/ldm/models/diffusion/ddim.py), you can substitute other samplers from the `ldm/models/diffusion/` module such as `PLMSSampler` or `DDPMSampler`. However, you must ensure the sampler supports conditioning on the concatenated 7-channel tensor structure required by the inpainting model configuration. The DDIM sampler is preferred for inpainting due to its deterministic nature and fewer sampling steps required for high-quality results.

### What is the expected resolution for input images and masks?

The inpainting model typically processes inputs at **512×512 pixels** resolution, though the `make_batch` function will automatically resize inputs if needed. For optimal results, provide images and masks at 512×512 resolution. The mask is downsampled via `torch.nn.functional.interpolate` to match the latent spatial dimensions (typically 64×64 for 512×512 inputs with an 8× downsampling factor) before being concatenated with the image encoding.