How to Optimize TRELLIS.2 Performance with CUDA Kernels and FlexGEMM

TRELLIS.2 achieves high-performance sparse voxel processing by combining custom CUDA kernels for rasterization and spatial hashing with the FlexGEMM library's GPU-accelerated sparse matrix multiplication primitives.

Optimizing 3D neural reconstruction pipelines requires careful tuning of both geometric processing and sparse convolution operations. The microsoft/TRELLIS.2 repository implements a hybrid architecture that leverages hand-optimized CUDA kernels for voxel grid generation while delegating matrix-heavy computations to FlexGEMM's specialized sparse kernels. Understanding how these components interact allows you to maximize throughput on modern NVIDIA GPUs.

Understanding TRELLIS.2's CUDA Architecture

Voxel Rasterization Kernels

The core geometric workload in TRELLIS.2 converts triangle meshes into sparse voxel grids using the rasterization engine in o-voxel/src/rasterize/rasterize.cu. These kernels exploit shared memory and warp-level primitives to maintain high occupancy during voxelization. The implementation processes triangle meshes in parallel batches, with each thread handling voxel coverage tests and atomic operations to populate the sparse grid.

Spatial Hashing and Indexing

Once voxels are generated, fast lookup becomes critical. The repository implements hash-based storage in o-voxel/src/hash/hash.cu, which accelerates spatial queries during both encoding and decoding phases. For memory-efficient storage and cache-friendly access patterns, TRELLIS.2 employs space-filling curve encodings located in o-voxel/src/serialize/z_order.cu and o-voxel/src/serialize/hilbert.cu. These utilities pack 3D coordinates into compact 32-bit representations that minimize memory bandwidth during grid traversal.

Integrating FlexGEMM for Sparse Operations

Configuring FlexGEMM Parameters

The Python interface in trellis2/modules/sparse/conv/conv_flex_gemm.py wraps FlexGEMM operators such as sparse_submanifold_conv3d and grid_sample_3d. To optimize performance, you can tune two critical parameters:

• algorithm: Set via flex_gemm.ops.spconv.set_algorithm() or passed directly to convolution calls. Use "auto" to let the library select the best kernel for the current sparsity pattern, or specify "spconv" or "grid_sample" for deterministic behavior.
• hashmap_ratio: Configure with flex_gemm.ops.spconv.set_hashmap_ratio(). Higher values (e.g., 1.5) improve collision handling in sparse convolutions at the cost of increased memory consumption.

Sparse Convolution Workflows

FlexGEMM handles the computational bottleneck in sparse 3D convolutions that process voxel features. Unlike dense operations that waste cycles on empty space, the library's kernels skip zero-valued voxels using the coordinate indices generated by the CUDA rasterization stage. This tight integration between the o-voxel CUDA backend and FlexGEMM frontend ensures that data stays on the GPU between voxelization and neural processing steps.

Performance Optimization Strategies

Kernel Launch Configuration

Tune the CUDA block size in o-voxel/src/rasterize/rasterize.cu to match your GPU's Streaming Multiprocessor (SM) count. For most NVIDIA architectures, blocks of 128-256 threads provide optimal occupancy. You can programmatically query the ideal configuration using cudaOccupancyMaxPotentialBlockSize before launching the mesh_to_flexible_dual_grid conversion.

Memory Layout and Bandwidth

Store voxel coordinates in the packed 32-bit format produced by the Z-order and Hilbert curve utilities to reduce memory bandwidth by up to 50% compared to raw float32 coordinates. Ensure tensors align to 256-byte boundaries for optimal memory coalescing when transferring data between the voxel grid and FlexGEMM operations.

FlexGEMM Algorithm Tuning

Experiment with the FLEX_GEMM_ALGO environment variable or runtime parameters in conv_flex_gemm.py. For highly sparse scenes (fewer than 1% occupancy), the "spconv" algorithm typically outperforms "grid_sample", while the reverse holds for moderately dense volumes. Set hashmap_ratio between 1.2 and 2.0 depending on your scene complexity—higher ratios prevent hash collisions that stall the GPU during feature aggregation.

Batch Processing

Group multiple meshes into a single batch before calling o_voxel.convert.mesh_to_flexible_dual_grid() to amortize kernel launch overhead. The rasterization kernels in rasterize.cu process batches more efficiently than sequential single-mesh conversions because they can maintain warps full of active threads across the entire batch.

Implementation Examples

The following examples demonstrate the complete optimization workflow, from mesh voxelization through FlexGEMM-accelerated sparse convolution.

Example 1: Basic Voxelization and Conversion Pipeline

import torch
import o_voxel
from flex_gemm.ops.spconv import sparse_submanifold_conv3d

# Load mesh data (verts: Nx3, faces: Mx3)

verts, faces = load_obj('example.obj')

# Step 1: Convert mesh to sparse voxel grid using optimized CUDA kernels

voxel_idx, dual_vert, intersect = o_voxel.convert.mesh_to_flexible_dual_grid(
    verts, faces, 
    voxel_size=0.01
)

# Step 2: Prepare features and weights for FlexGEMM processing

features = torch.randn(voxel_idx.shape[0], 64, device='cuda')
weight = torch.randn(64, 128, 3, 3, 3, device='cuda')

# Step 3: Apply sparse submanifold convolution

out = sparse_submanifold_conv3d(
    features, 
    voxel_idx, 
    weight,
    algorithm='auto',
    hashmap_ratio=1.5
)

# Step 4: Reconstruct mesh if needed

verts_rec, faces_rec = o_voxel.convert.flexible_dual_grid_to_mesh(
    voxel_idx, dual_vert, intersect
)

Example 2: Tuning Rasterization with Custom Block Sizes

import torch
import o_voxel

# Configure GPU for deterministic behavior

torch.backends.cuda.matmul.allow_tf32 = False
torch.cuda.set_device(0)

# Load geometry

verts, faces = load_obj('complex_scene.obj')

# Override default kernel launch parameters from rasterize.cu

# Higher block sizes (256) improve throughput for large meshes

voxel_idx, dual_vert, intersect = o_voxel.convert.mesh_to_flexible_dual_grid(
    verts, 
    faces,
    voxel_size=0.005,
    block_size=256  # Matches optimized occupancy in rasterize.cu

)

Example 3: End-to-End Optimized Pipeline

import torch
import o_voxel
import flex_gemm.ops.spconv as spconv

def render_optimized_mesh(mesh_path):
    verts, faces = load_obj(mesh_path)
    
    # Voxelize with optimal memory layout

    vox_idx, vox_vert, intersect = o_voxel.convert.mesh_to_flexible_dual_grid(
        verts, faces, 
        voxel_size=0.01
    )
    
    # Feature extraction with FlexGEMM

    feats = torch.randn(vox_idx.shape[0], 32, device='cuda')
    weight = torch.randn(32, 64, 3, 3, 3, device='cuda')
    
    conv_out = spconv.sparse_submanifold_conv3d(
        feats, 
        vox_idx, 
        weight,
        algorithm='auto',
        hashmap_ratio=1.2  # Balanced for medium-density scenes

    )
    
    # Reconstruct with dual grid conversion

    new_verts, new_faces = o_voxel.convert.flexible_dual_grid_to_mesh(
        vox_idx, vox_vert, intersect
    )
    
    return new_verts, new_faces, conv_out

Summary

• Custom CUDA kernels in o-voxel/src/rasterize/rasterize.cu and o-voxel/src/hash/hash.cu handle geometric conversion and spatial indexing with warp-level optimization.
• Space-filling curves (z_order.cu, hilbert.cu) compact 3D coordinates to minimize memory bandwidth during grid operations.
• FlexGEMM integration through trellis2/modules/sparse/conv/conv_flex_gemm.py accelerates sparse convolutions via tunable algorithms and hashmap ratios.
• Performance tuning requires aligning block sizes to GPU architectures, batching mesh processing, and selecting FlexGEMM algorithms based on scene sparsity.

Frequently Asked Questions

How does TRELLIS.2 handle memory efficiency in sparse voxel grids?

TRELLIS.2 implements hash-based storage in o-voxel/src/hash/hash.cu combined with space-filling curve encodings from z_order.cu and hilbert.cu. These components pack 3D coordinates into 32-bit integers that consume less memory than float32 tuples while maintaining cache-friendly access patterns during spatial lookups.

What is the optimal hashmap_ratio for FlexGEMM operations in TRELLIS.2?

The optimal hashmap_ratio depends on your scene density. For sparse scenes with scattered voxels, use ratios between 1.2 and 1.5 to prevent hash collisions. Denser volumes benefit from ratios up to 2.0, though this increases GPU memory consumption. Configure this parameter in conv_flex_gemm.py or via flex_gemm.ops.spconv.set_hashmap_ratio().

Can I modify the CUDA kernel block sizes in the rasterization stage?

Yes. While the default configurations in o-voxel/src/rasterize/rasterize.cu target 128-256 threads for general GPU compatibility, you can override these at the Python API level using the block_size parameter in mesh_to_flexible_dual_grid(). Use cudaOccupancyMaxPotentialBlockSize profiling to determine the ideal configuration for your specific GPU architecture.

How do I install FlexGEMM for TRELLIS.2 optimization?

Run the setup script with the --flexgemm flag to install FlexGEMM as a submodule and ensure CUDA toolchain compatibility. The installation process compiles the sparse convolution kernels against your system's CUDA version, enabling the optimized paths in trellis2/modules/sparse/conv/conv_flex_gemm.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}