# How img2img Inpainting and Outpainting Work in AUTOMATIC1111 Stable Diffusion WebUI

> Learn how img2img inpainting and outpainting function in AUTOMATIC1111 Stable Diffusion WebUI. Understand mask preparation and FFT-based noise matching for image generation.

- Repository: [AUTOMATIC1111/stable-diffusion-webui](https://github.com/AUTOMATIC1111/stable-diffusion-webui)
- Tags: deep-dive
- Published: 2026-02-24

---

**AUTOMATIC1111's WebUI handles img2img inpainting through five distinct modes (0-4) in [`modules/img2img.py`](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/main/modules/img2img.py) that prepare masks and images for `StableDiffusionProcessingImg2Img`, while outpainting extends this pipeline via the Outpainting mk2 script in [`scripts/outpainting_mk_2.py`](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/main/scripts/outpainting_mk_2.py) using FFT-based noise matching and iterative canvas expansion.**

The AUTOMATIC1111/stable-diffusion-webui repository provides a flexible image-to-image generation framework that supports both inpainting (editing specific regions) and outpainting (extending canvas boundaries) through the same underlying diffusion backend. Understanding how the **img2img inpainting outpainting** pipeline differentiates between these modes requires examining the dispatcher logic, mask preparation algorithms, and script-based extensions that manipulate the input latents before the model processes them.

## The img2img Mode Dispatcher

The core entry point for all image-to-image operations resides in **[modules/img2img.py](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/master/modules/img2img.py)**, specifically within the `img2img` function (lines 152‑180). This dispatcher receives a *mode* integer from the Gradio UI and routes the request through distinct preprocessing branches:

- **Mode 0**: Classic img2img without masking
- **Mode 1**: Sketch-to-image (treats the sketch as the base input)
- **Mode 2**: Inpainting with an uploaded mask bundled with the image
- **Mode 3**: Inpainting from a color sketch (mask generated via pixel comparison)
- **Mode 4**: Inpainting with a separately uploaded mask file

After selecting the appropriate branch, the dispatcher normalizes inputs using `images.fix_image` and constructs a **StableDiffusionProcessingImg2Img** object defined in **[modules/processing.py](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/master/modules/processing.py)**. This object encapsulates the image, mask, denoising strength, and inpainting-specific parameters before passing control to `modules.scripts.scripts_img2img.run`.

## Inpainting Implementation (Modes 2‑4)

Inpainting operations share a common preprocessing flow that prepares binary masks and injects region-specific parameters into the processing object.

### Mode 2: Uploaded Mask Inpainting

When `mode == 2`, the system expects `init_img_with_mask` containing both the base image and its corresponding mask. The helper `processing.create_binary_mask` (located in **modules/processing.py**) converts the provided mask into a pure black-and-white representation. The mask is then attached to the processing object as `p.mask`, while parameters like `mask_blur` and `inpainting_fill` populate the configuration (lines 200‑208).

### Mode 3: Color Sketch Inpainting

Mode 3 handles scenarios where users draw a color sketch over the original image. The code generates the mask dynamically by comparing pixels between the modified sketch and the original:

```python
mask = np.any(np.array(image) != np.array(orig), axis=-1)

```

This boolean mask undergoes post-processing with `ImageEnhance.Brightness` to soften edges and `ImageFilter.GaussianBlur` to ensure smooth transitions between painted and unpainted regions before binary conversion.

### Mode 4: Separate Mask File

For `mode == 4`, the mask arrives as a distinct upload via `init_mask_inpaint`. This path bypasses the bundled image+mask structure of Mode 2 but otherwise follows identical binary conversion and parameter injection procedures.

## Outpainting Architecture via Outpainting mk2

Unlike inpainting, **outpainting** is not a native mode of the `img2img` dispatcher. Instead, it operates as an img2img script found in **[scripts/outpainting_mk_2.py](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/master/scripts/outpainting_mk_2.py)**. The script's `show` method returns `True` only for img2img contexts, adding outpainting controls to the standard interface.

### Canvas Expansion and Sizing

The script calculates target dimensions that are multiples of 64 (the Stable Diffusion latent block size) based on user-specified pixel expansions for `left`, `right`, `up`, and `down` directions (lines 71‑86). It creates a white canvas with a black rectangle representing the original image position—this black region becomes the **mask** that protects existing content while the white surrounding area triggers generation.

### FFT-Based Noise Matching

To ensure seamless blending, `get_matched_noise` (lines 15‑118) generates a noise texture that matches the spectral statistics of the original image using Fast Fourier Transform (FFT) filtering. The function then applies histogram matching via `skimage.exposure.match_histograms` to align the noise distribution with the source image's color characteristics.

### Iterative Directional Expansion

The `expand` helper (lines 87‑155) processes each direction separately:

1. Creates intermediate image and mask pairs for the current expansion side
2. Configures the processing object with `p.do_not_save_samples = True`, `p.inpaint_full_res = False`, and `p.inpainting_fill = 1`
3. Calls `process_images(p)` to generate the new edge content
4. Pastes the generated patch back onto the main canvas

This loop repeats for each selected direction, respecting global `batch_size` and `n_iter` settings while temporarily suppressing intermediate sample saving to avoid clutter.

## Architectural Flow Summary

Both inpainting and outpainting ultimately converge on the same diffusion backend, differing only in how they prepare the `init_images` and `image_mask` tensors:

```

UI (Gradio) → img2img(mode) → modules/img2img.py
   ├─ Selects image/mask based on mode (0-4)
   ├─ Builds StableDiffusionProcessingImg2Img
   ├─ Executes scripts (e.g., Outpainting mk2)
   │      └─ Expands canvas, generates matched noise, calls process_images()
   └─ process_images() → Diffusion model → Processed result

```

The **Outpainting mk2** script manipulates the processing object's dimensions, mask, and noise parameters before delegating to the standard inpainting pipeline, while native modes handle mask preparation directly within the dispatcher.

## Practical Code Examples

### Triggering Inpainting Mode 2 Programmatically

To invoke inpainting with an uploaded mask via Python:

```python
from modules import img2img, processing
from PIL import Image

init_img = Image.open("photo.png")
mask = Image.open("mask.png")

images, js, info, comments = img2img.img2img(
    id_task="inpaint_test",
    request=None,
    mode=2,  # Inpainting with uploaded mask

    prompt="a medieval castle",
    negative_prompt="low quality",
    prompt_styles=[],
    init_img=None,
    sketch=None,
    init_img_with_mask={"image": init_img, "mask": mask},
    inpaint_color_sketch=None,
    inpaint_color_sketch_orig=None,
    init_img_inpaint=None,
    init_mask_inpaint=None,
    mask_blur=4,
    mask_alpha=0,
    inpainting_fill=1,
    n_iter=1,
    batch_size=1,
    cfg_scale=7.0,
    image_cfg_scale=1.0,
    denoising_strength=0.75,
    selected_scale_tab=0,
    height=512,
    width=512,
    scale_by=1.0,
    resize_mode=0,
    inpaint_full_res=False,
    inpaint_full_res_padding=0,
    inpainting_mask_invert=0,
    img2img_batch_input_dir="",
    img2img_batch_output_dir="",
    img2img_batch_inpaint_mask_dir="",
    override_settings_texts=[],
    img2img_batch_use_png_info=False,
    img2img_batch_png_info_props=[],
    img2img_batch_png_info_dir="",
    img2img_batch_source_type="",
    img2img_batch_upload=[],
    *[]
)

```

The `mode=2` argument selects the branch that extracts the mask from `init_img_with_mask` and applies `processing.create_binary_mask` automatically.

### Running Outpainting mk2 from Python

To reproduce the outpainting workflow programmatically:

```python
from modules import scripts, processing, shared
from PIL import Image

# Locate the Outpainting mk2 script instance

script = next(s for s in scripts.scripts_img2img.scripts 
              if s.title() == "Outpainting mk2")

# Configure the base processing object

p = processing.StableDiffusionProcessingImg2Img(
    sd_model=shared.sd_model,
    outpath_samples=shared.opts.outdir_samples,
    outpath_grids=shared.opts.outdir_grids,
    prompt="a sunrise over mountains",
    negative_prompt="low quality",
    styles=[],
    batch_size=1,
    n_iter=1,
    cfg_scale=7.0,
    width=512,
    height=512,
    init_images=[Image.open("center_crop.png")],
    mask=None,
    mask_blur=8,
    inpainting_fill=1,
    resize_mode=0,
    denoising_strength=0.8,
    image_cfg_scale=1.0,
    inpaint_full_res=False,
    inpaint_full_res_padding=0,
    inpainting_mask_invert=0,
    override_settings={}
)

# Execute outpainting with 128px expansion in all directions

processed = script.run(
    p, 
    None,  # script args placeholder

    pixels=128,
    mask_blur=8,
    direction=["left", "right", "up", "down"],
    noise_q=1.0,
    color_variation=0.05
)

outpainted_images = processed.images

```

The script internally modifies `p.init_images`, `p.image_mask`, and canvas dimensions before invoking `process_images(p)` for each expansion direction.

## Summary

- **Mode-based dispatching**: The `img2img` function in [`modules/img2img.py`](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/main/modules/img2img.py) uses integers 0-4 to select between standard img2img, sketch, and three distinct inpainting mask input methods.
- **Binary mask preparation**: All inpainting modes eventually produce binary masks via `processing.create_binary_mask`, with Mode 3 dynamically generating masks from color sketch deltas.
- **Script-based outpainting**: Outpainting extends the pipeline through [`scripts/outpainting_mk_2.py`](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/main/scripts/outpainting_mk_2.py), which expands canvases to 64-pixel multiples and uses FFT noise matching for seamless edge generation.
- **Unified backend**: Both techniques rely on `StableDiffusionProcessingImg2Img` and `process_images()`, differing only in input preparation rather than model architecture.
- **Programmatic access**: The entire workflow is accessible via Python by importing `modules.img2img` for native modes or invoking specific script `run()` methods for outpainting.

## Frequently Asked Questions

### What is the difference between inpainting and outpainting in AUTOMATIC1111?

**Inpainting** modifies existing regions within an image using masks (Modes 2‑4), while **outpainting** extends the canvas boundaries beyond the original image dimensions. Inpainting masks protect specific areas of the original image, whereas outpainting masks protect the entire original image while generating new content in the expanded white-space regions. Outpainting is implemented as a script ([`outpainting_mk_2.py`](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/main/outpainting_mk_2.py)) rather than a native img2img mode.

### How does the WebUI handle mask preparation for color sketch inpainting?

In Mode 3, the system compares the uploaded color sketch pixel-by-pixel against the original sketch using `np.any(np.array(image) != np.array(orig), axis=-1)` to identify changed regions. The resulting boolean mask is softened using `ImageEnhance.Brightness` and blurred with `ImageFilter.GaussianBlur` before binary conversion via `processing.create_binary_mask`, ensuring smooth transitions between edited and preserved areas.

### Why does outpainting require canvas sizes in multiples of 64?

Stable Diffusion operates on latent space representations where each spatial unit corresponds to 8×8 pixel blocks (64 pixels total per latent block). The Outpainting mk2 script enforces this alignment in lines 71‑86 to prevent latent dimension mismatches that would cause tensor shape errors during the convolution operations in the U-Net architecture.

### Can outpainting scripts be used with custom inpainting models?

Yes, the Outpainting mk2 script is model-agnostic and works with any checkpoint compatible with `StableDiffusionProcessingImg2Img`. The script manipulates the input image and mask tensors before the diffusion step, meaning specialized inpainting models (such as those trained with additional mask channels) will receive the correctly formatted inputs provided they are loaded as the active `shared.sd_model` before processing begins.