How to Implement Background Removal with U-2-Net or MODNet in ailia-models

Implement background removal in ailia-models by using the pre-trained U-2-Net or MODNet ONNX models with the ailia SDK, processing images through resize-normalization-inference pipelines available in background_removal/u2net/u2net.py and background_removal/modnet/modnet.py.

The axinc-ai/ailia-models repository provides production-ready implementations for background removal using deep learning. Both U-2-Net and MODNet are available as ONNX models wrapped by the ailia SDK, enabling high-performance alpha matting for images and video without requiring manual trimap creation.

Understanding the Background Removal Architecture

The repository offers two distinct approaches to background removal: U-2-Net for salient object detection and MODNet for trimap-free portrait matting. Both models follow a consistent inference pattern defined in background_removal/u2net/u2net.py and background_removal/modnet/modnet.py: automatic model download via model_utils.check_and_download_models, initialization with ailia.Net, pre-processing, inference with net.predict, and post-processing to generate the alpha matte.

Implementing Background Removal with U-2-Net

Model Architecture and Pre-processing

U-2-Net utilizes a nested U-structure encoder-decoder that aggregates multi-scale features through deep supervision. According to the source code in background_removal/u2net/u2net.py, the model accepts a default input size of 320×320 pixels and outputs a single-channel probability map (α mask). The script supports both the full-size model (u2net.onnx) and the lightweight variant (u2netp.onnx), selected via the model list configuration (lines 30-40).

U-2-Net Implementation Example

The following implementation mirrors the CLI logic found in u2net.py:

# u2net_demo.py

import ailia
from image_utils import imread
from u2net_utils import load_image, norm, save_result, transform
from model_utils import check_and_download_models

# ---- Configuration -------------------------------------------------

WEIGHT = "u2net.onnx"                # or u2netp.onnx for the small model

PROTO  = "u2net.onnx.prototxt"
REMOTE = "https://storage.googleapis.com/ailia-models/u2net/"

# ---- Model preparation ----------------------------------------------

check_and_download_models(WEIGHT, PROTO, REMOTE)
net = ailia.Net(PROTO, WEIGHT, env_id=0)

# ---- Inference -------------------------------------------------------

input_path  = "input.jpg"
output_path = "output.png"

# 1. Load & resize to network size (default 320×320)

img, h, w = load_image(input_path, scaled_size=(320, 320), rgb_mode=False)

# 2. Forward pass

pred = net.predict([img])[0][0, 0, :, :]          # shape H×W

# 3. Restore original resolution & save

save_result(pred, output_path, [h, w])
print("Result saved to:", output_path)

Key implementation details from the source:

• The load_image function in u2net_utils.py handles resizing and normalization
• Inference returns a tensor of shape 1×1×H×W, requiring extraction of the first channel
• The --composite flag (lines 44-48) adds an alpha channel to the original BGR image for direct compositing

Implementing Background Removal with MODNet

Model Architecture and Scaling Requirements

MODNet implements a lightweight encoder-decoder with three branches (detail, semantic, and fusion) that predicts a high-resolution matte without requiring a trimap. As implemented in background_removal/modnet/modnet.py, the model uses a reference height of 512 pixels (INFERENCE_HEIGHT = 512, line 41) and requires input dimensions to be multiples of 32. The modnet_utils.get_scale_factor function (lines 3-22) calculates the scaling factor to ensure this alignment.

MODNet Implementation Example

This code reproduces the workflow from modnet.py:

# modnet_demo.py

import ailia
import cv2, numpy as np
from modnet_utils import get_scale_factor
from model_utils import check_and_download_models
from image_utils import imread

# ---- Configuration -------------------------------------------------

WEIGHT = "modnet.opt.onnx"
PROTO  = "modnet.opt.onnx.prototxt"
REMOTE = "https://storage.googleapis.com/ailia-models/modnet/"

# ---- Model preparation ----------------------------------------------

check_and_download_models(WEIGHT, PROTO, REMOTE)
net = ailia.Net(PROTO, WEIGHT, env_id=0)

# ---- Inference -------------------------------------------------------

img_path = "input.jpg"
out_path = "output.png"

raw = imread(img_path)               # BGR

rgb = cv2.cvtColor(raw, cv2.COLOR_BGR2RGB).astype("float32")
rgb = (rgb - 127.5) / 127.5          # normalise to [-1, 1]

h, w, _ = rgb.shape
sx, sy = get_scale_factor(h, w, 512) # reference height (line 41)

# Resize while keeping factor multiples of 32

rgb_resized = cv2.resize(rgb, None, fx=sx, fy=sy, interpolation=cv2.INTER_AREA)

# Prepare NCHW tensor

blob = np.transpose(rgb_resized, (2, 0, 1))[np.newaxis, ...].astype("float32")

# Forward

pred = net.predict(blob)[0]
matte = (np.squeeze(pred) * 255).astype("uint8")
matte = cv2.resize(matte, (w, h), interpolation=cv2.INTER_AREA)

# Optional compositing

rgba = cv2.cvtColor(raw, cv2.COLOR_BGR2BGRA)
rgba[:, :, 3] = matte
cv2.imwrite(out_path, rgba)
print("Result saved to:", out_path)

Critical differences from U-2-Net:

• Normalization: MODNet requires scaling to [-1, 1] rather than standard ImageNet normalization
• Soft matte: The output is a float 0-1 mask scaled to 0-255 (line 79), providing smoother edges than U-2-Net's binary-like output
• Dimension constraints: Width and height must be divisible by 32, enforced by get_scale_factor

Comparing U-2-Net and MODNet for Background Removal

Both scripts expose a consistent CLI (--input, --savepath, --video, --composite) using the generic arg_utils helper, but differ in their computer vision approach:

• U-2-Net: Optimized for general salient object detection with a nested U-structure; best for arbitrary objects against varied backgrounds
• MODNet: Specialized for portrait matting with trimap-free inference; ideal for human subjects requiring fine hair detail preservation

Summary

• Background removal in ailia-models is implemented via ONNX models wrapped by the ailia.Net class
• U-2-Net (background_removal/u2net/u2net.py) processes 320×320 inputs using deep supervision for binary mask generation
• MODNet (background_removal/modnet/modnet.py) processes 512×736 inputs with 32-pixel alignment constraints for soft alpha matting
• Both pipelines use model_utils.check_and_download_models for automatic weight management and support video processing via recognize_from_video

Frequently Asked Questions

What is the difference between U-2-Net and MODNet for background removal?

U-2-Net performs salient object detection outputting a binary mask suitable for general objects, while MODNet performs trimap-free portrait matting producing a soft alpha matte optimized for human subjects. U-2-Net uses a nested U-structure with 320×320 inputs, whereas MODNet employs a three-branch architecture requiring 32-pixel aligned inputs at 512×736 resolution.

How do I process video files for background removal?

Both u2net.py and modnet.py implement recognize_from_video functions that capture frames using OpenCV, apply the same pre-processing pipeline as images, and optionally composite the matte onto the original frame in real-time. Use the --video CLI argument to specify the input video path.

Can I run these models on CPU-only systems?

Yes. The ailia.Net initialization accepts an env_id parameter where 0 typically selects CPU inference. The ONNX models are optimized for the ailia SDK's CPU backend, though GPU acceleration is available when CUDA or Metal drivers are present.

Where does the code download the model weights?

The check_and_download_models function in util/model_utils.py automatically retrieves ONNX weights and prototxt files from Google Cloud Storage URLs (e.g., https://storage.googleapis.com/ailia-models/u2net/) if they are not present in the local working directory.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}