How img2img Inpainting and Outpainting Work in AUTOMATIC1111 Stable Diffusion WebUI

AUTOMATIC1111's WebUI handles img2img inpainting through five distinct modes (0-4) in modules/img2img.py that prepare masks and images for StableDiffusionProcessingImg2Img, while outpainting extends this pipeline via the Outpainting mk2 script in scripts/outpainting_mk_2.py using FFT-based noise matching and iterative canvas expansion.

The AUTOMATIC1111/stable-diffusion-webui repository provides a flexible image-to-image generation framework that supports both inpainting (editing specific regions) and outpainting (extending canvas boundaries) through the same underlying diffusion backend. Understanding how the img2img inpainting outpainting pipeline differentiates between these modes requires examining the dispatcher logic, mask preparation algorithms, and script-based extensions that manipulate the input latents before the model processes them.

The img2img Mode Dispatcher

The core entry point for all image-to-image operations resides in modules/img2img.py, specifically within the img2img function (lines 152‑180). This dispatcher receives a mode integer from the Gradio UI and routes the request through distinct preprocessing branches:

• Mode 0: Classic img2img without masking
• Mode 1: Sketch-to-image (treats the sketch as the base input)
• Mode 2: Inpainting with an uploaded mask bundled with the image
• Mode 3: Inpainting from a color sketch (mask generated via pixel comparison)
• Mode 4: Inpainting with a separately uploaded mask file

After selecting the appropriate branch, the dispatcher normalizes inputs using images.fix_image and constructs a StableDiffusionProcessingImg2Img object defined in modules/processing.py. This object encapsulates the image, mask, denoising strength, and inpainting-specific parameters before passing control to modules.scripts.scripts_img2img.run.

Inpainting Implementation (Modes 2‑4)

Inpainting operations share a common preprocessing flow that prepares binary masks and injects region-specific parameters into the processing object.

Mode 2: Uploaded Mask Inpainting

When mode == 2, the system expects init_img_with_mask containing both the base image and its corresponding mask. The helper processing.create_binary_mask (located in modules/processing.py) converts the provided mask into a pure black-and-white representation. The mask is then attached to the processing object as p.mask, while parameters like mask_blur and inpainting_fill populate the configuration (lines 200‑208).

Mode 3: Color Sketch Inpainting

Mode 3 handles scenarios where users draw a color sketch over the original image. The code generates the mask dynamically by comparing pixels between the modified sketch and the original:

mask = np.any(np.array(image) != np.array(orig), axis=-1)

This boolean mask undergoes post-processing with ImageEnhance.Brightness to soften edges and ImageFilter.GaussianBlur to ensure smooth transitions between painted and unpainted regions before binary conversion.

Mode 4: Separate Mask File

For mode == 4, the mask arrives as a distinct upload via init_mask_inpaint. This path bypasses the bundled image+mask structure of Mode 2 but otherwise follows identical binary conversion and parameter injection procedures.

Outpainting Architecture via Outpainting mk2

Unlike inpainting, outpainting is not a native mode of the img2img dispatcher. Instead, it operates as an img2img script found in scripts/outpainting_mk_2.py. The script's show method returns True only for img2img contexts, adding outpainting controls to the standard interface.

Canvas Expansion and Sizing

The script calculates target dimensions that are multiples of 64 (the Stable Diffusion latent block size) based on user-specified pixel expansions for left, right, up, and down directions (lines 71‑86). It creates a white canvas with a black rectangle representing the original image position—this black region becomes the mask that protects existing content while the white surrounding area triggers generation.

FFT-Based Noise Matching

To ensure seamless blending, get_matched_noise (lines 15‑118) generates a noise texture that matches the spectral statistics of the original image using Fast Fourier Transform (FFT) filtering. The function then applies histogram matching via skimage.exposure.match_histograms to align the noise distribution with the source image's color characteristics.

Iterative Directional Expansion

The expand helper (lines 87‑155) processes each direction separately:

1. Creates intermediate image and mask pairs for the current expansion side
2. Configures the processing object with p.do_not_save_samples = True, p.inpaint_full_res = False, and p.inpainting_fill = 1
3. Calls process_images(p) to generate the new edge content
4. Pastes the generated patch back onto the main canvas

This loop repeats for each selected direction, respecting global batch_size and n_iter settings while temporarily suppressing intermediate sample saving to avoid clutter.

Architectural Flow Summary

Both inpainting and outpainting ultimately converge on the same diffusion backend, differing only in how they prepare the init_images and image_mask tensors:

UI (Gradio) → img2img(mode) → modules/img2img.py
   ├─ Selects image/mask based on mode (0-4)
   ├─ Builds StableDiffusionProcessingImg2Img
   ├─ Executes scripts (e.g., Outpainting mk2)
   │      └─ Expands canvas, generates matched noise, calls process_images()
   └─ process_images() → Diffusion model → Processed result

The Outpainting mk2 script manipulates the processing object's dimensions, mask, and noise parameters before delegating to the standard inpainting pipeline, while native modes handle mask preparation directly within the dispatcher.

Practical Code Examples

Triggering Inpainting Mode 2 Programmatically

To invoke inpainting with an uploaded mask via Python:

from modules import img2img, processing
from PIL import Image

init_img = Image.open("photo.png")
mask = Image.open("mask.png")

images, js, info, comments = img2img.img2img(
    id_task="inpaint_test",
    request=None,
    mode=2,  # Inpainting with uploaded mask

    prompt="a medieval castle",
    negative_prompt="low quality",
    prompt_styles=[],
    init_img=None,
    sketch=None,
    init_img_with_mask={"image": init_img, "mask": mask},
    inpaint_color_sketch=None,
    inpaint_color_sketch_orig=None,
    init_img_inpaint=None,
    init_mask_inpaint=None,
    mask_blur=4,
    mask_alpha=0,
    inpainting_fill=1,
    n_iter=1,
    batch_size=1,
    cfg_scale=7.0,
    image_cfg_scale=1.0,
    denoising_strength=0.75,
    selected_scale_tab=0,
    height=512,
    width=512,
    scale_by=1.0,
    resize_mode=0,
    inpaint_full_res=False,
    inpaint_full_res_padding=0,
    inpainting_mask_invert=0,
    img2img_batch_input_dir="",
    img2img_batch_output_dir="",
    img2img_batch_inpaint_mask_dir="",
    override_settings_texts=[],
    img2img_batch_use_png_info=False,
    img2img_batch_png_info_props=[],
    img2img_batch_png_info_dir="",
    img2img_batch_source_type="",
    img2img_batch_upload=[],
    *[]
)

The mode=2 argument selects the branch that extracts the mask from init_img_with_mask and applies processing.create_binary_mask automatically.

Running Outpainting mk2 from Python

To reproduce the outpainting workflow programmatically:

from modules import scripts, processing, shared
from PIL import Image

# Locate the Outpainting mk2 script instance

script = next(s for s in scripts.scripts_img2img.scripts 
              if s.title() == "Outpainting mk2")

# Configure the base processing object

p = processing.StableDiffusionProcessingImg2Img(
    sd_model=shared.sd_model,
    outpath_samples=shared.opts.outdir_samples,
    outpath_grids=shared.opts.outdir_grids,
    prompt="a sunrise over mountains",
    negative_prompt="low quality",
    styles=[],
    batch_size=1,
    n_iter=1,
    cfg_scale=7.0,
    width=512,
    height=512,
    init_images=[Image.open("center_crop.png")],
    mask=None,
    mask_blur=8,
    inpainting_fill=1,
    resize_mode=0,
    denoising_strength=0.8,
    image_cfg_scale=1.0,
    inpaint_full_res=False,
    inpaint_full_res_padding=0,
    inpainting_mask_invert=0,
    override_settings={}
)

# Execute outpainting with 128px expansion in all directions

processed = script.run(
    p, 
    None,  # script args placeholder

    pixels=128,
    mask_blur=8,
    direction=["left", "right", "up", "down"],
    noise_q=1.0,
    color_variation=0.05
)

outpainted_images = processed.images

The script internally modifies p.init_images, p.image_mask, and canvas dimensions before invoking process_images(p) for each expansion direction.

Summary

• Mode-based dispatching: The img2img function in modules/img2img.py uses integers 0-4 to select between standard img2img, sketch, and three distinct inpainting mask input methods.
• Binary mask preparation: All inpainting modes eventually produce binary masks via processing.create_binary_mask, with Mode 3 dynamically generating masks from color sketch deltas.
• Script-based outpainting: Outpainting extends the pipeline through scripts/outpainting_mk_2.py, which expands canvases to 64-pixel multiples and uses FFT noise matching for seamless edge generation.
• Unified backend: Both techniques rely on StableDiffusionProcessingImg2Img and process_images(), differing only in input preparation rather than model architecture.
• Programmatic access: The entire workflow is accessible via Python by importing modules.img2img for native modes or invoking specific script run() methods for outpainting.

Frequently Asked Questions

What is the difference between inpainting and outpainting in AUTOMATIC1111?

Inpainting modifies existing regions within an image using masks (Modes 2‑4), while outpainting extends the canvas boundaries beyond the original image dimensions. Inpainting masks protect specific areas of the original image, whereas outpainting masks protect the entire original image while generating new content in the expanded white-space regions. Outpainting is implemented as a script (outpainting_mk_2.py) rather than a native img2img mode.

How does the WebUI handle mask preparation for color sketch inpainting?

In Mode 3, the system compares the uploaded color sketch pixel-by-pixel against the original sketch using np.any(np.array(image) != np.array(orig), axis=-1) to identify changed regions. The resulting boolean mask is softened using ImageEnhance.Brightness and blurred with ImageFilter.GaussianBlur before binary conversion via processing.create_binary_mask, ensuring smooth transitions between edited and preserved areas.

Why does outpainting require canvas sizes in multiples of 64?

Stable Diffusion operates on latent space representations where each spatial unit corresponds to 8×8 pixel blocks (64 pixels total per latent block). The Outpainting mk2 script enforces this alignment in lines 71‑86 to prevent latent dimension mismatches that would cause tensor shape errors during the convolution operations in the U-Net architecture.

Can outpainting scripts be used with custom inpainting models?

Yes, the Outpainting mk2 script is model-agnostic and works with any checkpoint compatible with StableDiffusionProcessingImg2Img. The script manipulates the input image and mask tensors before the diffusion step, meaning specialized inpainting models (such as those trained with additional mask channels) will receive the correctly formatted inputs provided they are loaded as the active shared.sd_model before processing begins.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}