How to Generate PBR Textures for Existing 3D Meshes Using Trellis2TexturingPipeline

The Trellis2TexturingPipeline is a high-level wrapper that converts any input mesh and reference image into a fully textured PBR-ready model by encoding geometry into sparse latent representations, sampling texture latents via a flow-based diffusion model, and rasterizing the results into base-color, metallic, roughness, and alpha maps.

The microsoft/TRELLIS.2 repository provides this production-ready implementation for automated texture synthesis. It bridges the gap between raw untextured geometry and production-ready assets through a modular ten-stage pipeline defined in trellis2/pipelines/trellis2_texturing.py.

Pipeline Architecture Overview

The Trellis2TexturingPipeline orchestrates ten distinct stages, each implemented as a specific method with precise line references in the source code.

1. Model Initialization The from_pretrained method loads model weights, sampler configurations, and normalization statistics from the Hugging Face repository microsoft/TRELLIS.2-4B.

2. Device Allocation The to method (lines 98-105) manages device placement. In low-VRAM mode, heavy sub-modules transfer to CUDA only when needed and return to CPU immediately after computation.

3. Mesh Preprocessing The preprocess_mesh method (lines 106-120) centers the geometry, normalizes its extent to ±0.5, and swaps Y/Z axes to ensure GLB compatibility.

4. Image Preprocessing The preprocess_image method (lines 122-158) rescales input, removes backgrounds using a Rembg model (when alpha is absent), crops to the tightest foreground bounding box, and premultiplies alpha values.

5. Conditioning Construction The get_cond method (lines 159-181) extracts image features via a conditioning encoder and prepares negative-conditioning tensors for classifier-free guidance.

6. Shape Latent Encoding The encode_shape_slat method (lines 182-226) converts the mesh into a flexible dual-grid representation, builds a SparseTensor, and processes it through the shape-slat encoder.

7. Texture Latent Sampling The sample_tex_slat method (lines 227-267) normalizes the shape latent, adds Gaussian noise, and samples texture representations using the configured flow model sampler.

8. Texture Decoding The decode_tex_slat method (lines 268-286) passes sampled latents through the texture-slat decoder to produce voxel grids storing PBR attributes.

9. Rasterization and Packing The postprocess_mesh method (lines 287-371) projects voxel grids onto mesh UVs using nvdiffrast, fills missing texels via OpenCV inpainting, and constructs a PBRMaterial object.

10. Output Generation The run method (lines 374-408) returns a trimesh.Trimesh object complete with UV coordinates and a TextureVisuals instance ready for export.

Step-by-Step Implementation

Loading the Pretrained Pipeline

Instantiate the pipeline from the official repository and move it to your target device:

import torch
from trellis2.pipelines import Trellis2TexturingPipeline

pipeline = Trellis2TexturingPipeline.from_pretrained(
    "microsoft/TRELLIS.2-4B",
    config_file="texturing_pipeline.json"
)

# For GPU acceleration

pipeline.cuda()

# For CPU or low-VRAM scenarios

# pipeline.to(torch.device("cpu"))

Preprocessing Input Assets

Load existing geometry and reference imagery using standard libraries. The pipeline accepts any format supported by trimesh (PLY, OBJ, GLB):

import trimesh
from PIL import Image

mesh_path = "assets/example_texturing/the_forgotten_knight.ply"
image_path = "assets/example_texturing/image.webp"

mesh = trimesh.load(mesh_path)
image = Image.open(image_path)

Executing the Texturing Pipeline

Invoke the run method with control parameters for reproducibility and quality:

textured_mesh = pipeline.run(
    mesh,
    image,
    seed=123,
    tex_slat_sampler_params={"num_steps": 50},
    preprocess_image=True,
    resolution=1024,
    texture_size=2048
)

Key parameters include:

• seed: Ensures reproducible stochastic sampling
• tex_slat_sampler_params: Configures diffusion steps and sampling behavior
• resolution: Controls internal voxel grid size (512 or 1024)
• texture_size: Sets final output texture dimensions (e.g., 2048×2048)

Exporting Textured Meshes

Save the result with embedded WebP-compressed textures:

output_path = "textured_output.glb"
textured_mesh.export(output_path, extension_webp=True)

Complete Implementation Example

The following end-to-end script mirrors the official example_texturing.py while demonstrating all configurable options:

import os
import torch
import trimesh
from PIL import Image
from trellis2.pipelines import Trellis2TexturingPipeline

# -------------------------------------------------

# 1️⃣ Load the pretrained pipeline

# -------------------------------------------------

pipeline = Trellis2TexturingPipeline.from_pretrained(
    "microsoft/TRELLIS.2-4B",
    config_file="texturing_pipeline.json"
)

# Choose device (GPU for speed, CPU for low‑VRAM)

pipeline.cuda()  # or pipeline.to(torch.device("cpu"))

# -------------------------------------------------

# 2️⃣ Load your mesh and reference image

# -------------------------------------------------

mesh_path = "assets/example_texturing/the_forgotten_knight.ply"
image_path = "assets/example_texturing/image.webp"

mesh = trimesh.load(mesh_path)
image = Image.open(image_path)

# -------------------------------------------------

# 3️⃣ Run the pipeline

# -------------------------------------------------

textured_mesh = pipeline.run(
    mesh,
    image,
    seed=123,
    tex_slat_sampler_params={"num_steps": 50},
    preprocess_image=True,
    resolution=1024,
    texture_size=2048
)

# -------------------------------------------------

# 4️⃣ Export the textured mesh

# -------------------------------------------------

output_path = "textured_output.glb"
textured_mesh.export(output_path, extension_webp=True)
print(f"Saved textured mesh to {output_path}")

Technical Deep Dive

Dual-Grid and Sparse Tensor Representation

The pipeline converts input meshes using o_voxel.convert.mesh_to_flexible_dual_grid to create sparse representations where each voxel stores dual vertices capturing geometric detail. These feed into SparseTensor objects defined in trellis2/modules/sparse, enabling efficient sparse convolutions throughout the encoding stages.

Flow-Based Sampling and Conditioning

Texture generation employs a normalizing-flow model (tex_slat_flow_model_*) integrated with the samplers.Sampler class. The get_cond method concatenates image and shape-latent conditioning vectors, then performs stochastic denoising to produce coherent PBR patterns.

Rasterization and Inpainting

The final projection uses nvdiffrast for UV rasterization, with flex_gemm.ops.grid_sample.grid_sample_3d handling trilinear interpolation from 3D voxel space to 2D texture coordinates. OpenCV inpainting repairs missing texels before the postprocess_mesh method packs channels into the final PBRMaterial.

Memory Optimization Strategies

When low_vram=True (the default), the pipeline keeps heavy modules such as flow models and image encoders on CPU, transferring them to GPU only during active computation. This selective device migration occurs automatically within each pipeline stage, keeping VRAM consumption manageable while processing high-resolution textures.

Summary

• Trellis2TexturingPipeline provides end-to-end PBR texture generation via trellis2/pipelines/trellis2_texturing.py with ten modular stages
• SparseTensor representations and dual-grid conversions enable efficient processing of complex geometries at high resolution
• The run method orchestrates preprocessing, encoding, sampling, and rasterization through a single interface
• Low-VRAM mode automatically manages device memory by cycling weights between CPU and GPU during processing
• Output meshes include UV coordinates and TextureVisuals compatible with GLB, OBJ, and other standard formats

Frequently Asked Questions

What input mesh formats does Trellis2TexturingPipeline support?

The pipeline accepts any format supported by the trimesh library, including PLY, OBJ, GLB, and STL. The preprocess_mesh method automatically handles coordinate system conversion (Y/Z axis swapping) and normalization to ensure compatibility with internal voxelization requirements.

How does the pipeline handle images without transparent backgrounds?

The preprocess_image method automatically detects missing alpha channels and invokes a Rembg model to remove backgrounds. It then crops to the tightest foreground bounding box and premultiplies alpha values, preparing the image for conditioning regardless of the original background state.

What is the difference between resolution and texture_size parameters?

The resolution parameter (512 or 1024) controls the internal voxel grid size used during shape encoding in encode_shape_slat and texture sampling in sample_tex_slat. The texture_size parameter (e.g., 2048) determines the final output texture dimensions rasterized onto mesh UVs during postprocess_mesh. Higher resolution values capture more geometric detail but require additional VRAM during processing.

Can I run Trellis2TexturingPipeline on CPU-only systems?

Yes. While GPU acceleration via pipeline.cuda() is recommended for performance, the pipeline supports CPU inference through pipeline.to(torch.device("cpu")). The low-VRAM mode further reduces memory requirements by keeping inactive model weights on CPU, though processing times will increase significantly without CUDA acceleration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}