How Background Removal Preprocessing Works in TRELLIS.2

Background removal preprocessing in TRELLIS.2 automatically isolates foreground objects using the BiRefNet segmentation model, resizing inputs to 1024px, generating alpha masks, and cropping to centered bounding boxes before 3D generation.

The preprocess_image method in the microsoft/TRELLIS.2 repository handles this critical preprocessing step before feeding images to the 3D generation models. Implemented identically in both Trellis2ImageTo3DPipeline and Trellis2TexturingPipeline, this pipeline relies on a configurable background-removal model that defaults to the BiRefNet implementation.

The Preprocessing Pipeline Architecture

Background removal preprocessing occurs in the preprocess_image method found in trellis2/pipelines/trellis2_image_to_3d.py and trellis2/pipelines/trellis2_texturing.py. Both pipelines instantiate a rembg_model attribute that defaults to the BiRefNet model loaded from Hugging Face, though the architecture supports swapping in alternative segmentation implementations.

The process operates on PIL.Image.Image inputs and returns processed RGB images with transparent or black backgrounds where the original background pixels were removed. This ensures that 3D generation models receive clean foreground objects without distracting background elements.

Six-Step Background Removal Process

The preprocess_image method executes a deterministic sequence to extract foreground objects from input photographs.

1. Alpha Channel Detection

The pipeline first checks if the input image already contains a usable mask. In trellis2/pipelines/trellis2_image_to_3d.py (lines 31-33), the code scans the alpha channel:

alpha = np.array(input)[:, :, 3]
has_alpha = np.any(alpha != 255)

If has_alpha evaluates to True—meaning any pixel has an alpha value not equal to 255—the image is treated as pre-masked and background removal is skipped entirely.

2. Efficiency Resizing

For performance optimization, the image is downscaled to a maximum dimension of 1024 pixels while preserving aspect ratio using Image.Resampling.LANCZOS (lines 38-40):

scale = min(1, 1024 / max_size)
new_size = (int(input.width * scale), int(input.height * scale))
input = input.resize(new_size, Image.Resampling.LANCZOS)

This resizing occurs before segmentation to reduce VRAM usage during model inference.

3. BiRefNet Segmentation

When no alpha channel exists, the RGB image is passed to the rembg_model running on the pipeline's device (CPU or CUDA). The call at lines 44-48 executes:

output = self.rembg_model(input)

The model returns an RGBA image where the newly generated alpha channel encodes the foreground mask, with white representing the object and black representing the background.

4. Foreground Bounding Box Extraction

The method extracts the foreground region by identifying pixels where alpha exceeds 80% intensity (lines 50-55):

bbox = np.argwhere(alpha > 0.8 * 255)
bbox = [bbox.min(axis=0)[1], bbox.min(axis=0)[0], 
        bbox.max(axis=0)[1] + 1, bbox.max(axis=0)[0] + 1]
output = output.crop(bbox)

This creates a centered square crop containing only the detected object, removing excess transparent padding.

5. RGB Premultiplication

To ensure proper blending in downstream 3D models, the cropped RGBA image undergoes premultiplication (lines 56-60):

output = np.array(output).astype(np.float32) / 255
output = output[:, :, :3] * output[:, :, 3:4]
output = (output * 255).astype(np.uint8)
output = Image.fromarray(output, mode='RGB')

This mathematical operation multiplies the RGB channels by the alpha mask, converting transparent regions to black and ensuring the foreground object blends correctly with 3D rendering backgrounds.

6. Output Generation

The final Image.Image object containing only the foreground object is returned at line 61, ready for conditioning the 3D generation networks.

The BiRefNet Model Implementation

The default background removal model resides in trellis2/pipelines/rembg/BiRefNet.py and implements a complete segmentation workflow.

Model Loading and Configuration

The BiRefNet class loads a pretrained AutoModelForImageSegmentation from Hugging Face (lines 9-13):

from transformers import AutoModelForImageSegmentation
self.model = AutoModelForImageSegmentation.from_pretrained(
    "ZhengPeng7/BiRefNet", trust_remote_code=True
)

Inference Pipeline

The model processes inputs through a standardized transformation pipeline (lines 14-19) including Resize to 1024×1024, ToTensor, and Normalize operations. During the forward pass (lines 35-40), the model generates a prediction tensor that is converted to a PIL mask via transforms.ToPILImage()(pred).

Alpha Channel Integration

The final step inserts the generated mask into the original image's alpha channel (lines 40-41):

image.putalpha(mask)

This produces the RGBA output that the preprocessing pipeline expects for subsequent bounding box extraction and cropping operations.

Memory Optimization Strategies

In low-VRAM environments, the pipeline implements dynamic device management to minimize GPU memory usage. The rembg_model temporarily moves to the inference device (self.rembg_model.to(self.device)) for segmentation and returns to CPU afterward. This prevents the BiRefNet parameters from permanently occupying GPU memory during the full 3D generation workflow, allowing the system to run on hardware with limited VRAM.

Practical Code Examples

Using the Image-to-3D Pipeline

from trellis2.pipelines.trellis2_image_to_3d import Trellis2ImageTo3DPipeline
from PIL import Image

# Load pretrained pipeline with BiRefNet background removal

pipeline = Trellis2ImageTo3DPipeline.from_pretrained(
    path="microsoft/trellis2-pretrained",
    config_file="pipeline.json"
)

# Optional: Move models to GPU

pipeline.to("cuda")

# Load RGB photograph

input_img = Image.open("photo.jpg")

# Background removal preprocessing happens automatically

foreground = pipeline.preprocess_image(input_img)
foreground.save("photo_fg.png")

Direct BiRefNet Invocation

from trellis2.pipelines.rembg.BiRefNet import BiRefNet
from PIL import Image

# Initialize background removal model

rembg = BiRefNet()
rembg.to("cuda")

# Process image directly

img = Image.open("portrait.jpg")
rgba_output = rembg(img)  # Returns RGBA with segmentation mask

rgba_output.save("portrait_fg.png")

Summary

• TRELLIS.2 implements background removal preprocessing in trellis2/pipelines/trellis2_image_to_3d.py and trellis2/pipelines/trellis2_texturing.py via the preprocess_image method.
• The pipeline defaults to BiRefNet segmentation but supports configurable replacement models through the rembg_model interface.
• Preprocessing includes alpha channel detection, 1024px resizing, foreground bounding box extraction (alpha > 0.8 threshold), and RGB premultiplication.
• The system optimizes for low-VRAM scenarios by temporarily moving the segmentation model to GPU only during inference.
• All operations preserve aspect ratio and center-crop results to provide consistent inputs for 3D generation models.

Frequently Asked Questions

What segmentation model does TRELLIS.2 use for background removal?

TRELLIS.2 uses BiRefNet (Bilateral Reference Network) as the default background removal model, loaded from the ZhengPeng7/BiRefNet Hugging Face repository. The model is instantiated in trellis2/pipelines/rembg/BiRefNet.py and resizes inputs to 1024×1024 resolution before generating foreground masks.

Does TRELLIS.2 skip background removal if the image already has transparency?

Yes. The preprocess_image method detects existing alpha channels by checking if any pixel value differs from 255 in the fourth channel. According to lines 31-33 of trellis2/pipelines/trellis2_image_to_3d.py, if np.any(alpha != 255) evaluates to True, the pipeline treats the image as pre-masked and bypasses the BiRefNet segmentation step entirely.

How does TRELLIS.2 handle limited GPU memory during preprocessing?

The pipeline implements temporary device migration for low-VRAM systems. The rembg_model moves to the active compute device (CUDA) immediately before inference and returns to CPU afterward, preventing the segmentation model from permanently occupying GPU memory during the full 3D generation workflow.

Can I replace the default BiRefNet model with custom background removal?

Yes. The pipeline accepts a configurable rembg_model parameter during initialization. Any model implementing the __call__(image: PIL.Image.Image) -> PIL.Image.Image interface that returns RGBA images with valid alpha channels can replace the default BiRefNet implementation in both Trellis2ImageTo3DPipeline and Trellis2TexturingPipeline.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}