# How to Optimize TRELLIS.2 Performance with CUDA Kernels and FlexGEMM

> Optimize TRELLIS.2 performance using custom CUDA kernels and FlexGEMM. Accelerate sparse voxel processing with GPU-powered sparse matrix multiplication.

- Repository: [Microsoft/TRELLIS.2](https://github.com/microsoft/TRELLIS.2)
- Tags: performance
- Published: 2026-08-03

---

**TRELLIS.2 achieves high-performance sparse voxel processing by combining custom CUDA kernels for rasterization and spatial hashing with the FlexGEMM library's GPU-accelerated sparse matrix multiplication primitives.**

Optimizing 3D neural reconstruction pipelines requires careful tuning of both geometric processing and sparse convolution operations. The microsoft/TRELLIS.2 repository implements a hybrid architecture that leverages hand-optimized CUDA kernels for voxel grid generation while delegating matrix-heavy computations to FlexGEMM's specialized sparse kernels. Understanding how these components interact allows you to maximize throughput on modern NVIDIA GPUs.

## Understanding TRELLIS.2's CUDA Architecture

### Voxel Rasterization Kernels

The core geometric workload in TRELLIS.2 converts triangle meshes into sparse voxel grids using the rasterization engine in `o-voxel/src/rasterize/rasterize.cu`. These kernels exploit shared memory and warp-level primitives to maintain high occupancy during voxelization. The implementation processes triangle meshes in parallel batches, with each thread handling voxel coverage tests and atomic operations to populate the sparse grid.

### Spatial Hashing and Indexing

Once voxels are generated, fast lookup becomes critical. The repository implements hash-based storage in `o-voxel/src/hash/hash.cu`, which accelerates spatial queries during both encoding and decoding phases. For memory-efficient storage and cache-friendly access patterns, TRELLIS.2 employs space-filling curve encodings located in `o-voxel/src/serialize/z_order.cu` and `o-voxel/src/serialize/hilbert.cu`. These utilities pack 3D coordinates into compact 32-bit representations that minimize memory bandwidth during grid traversal.

## Integrating FlexGEMM for Sparse Operations

### Configuring FlexGEMM Parameters

The Python interface in [`trellis2/modules/sparse/conv/conv_flex_gemm.py`](https://github.com/microsoft/TRELLIS.2/blob/main/trellis2/modules/sparse/conv/conv_flex_gemm.py) wraps FlexGEMM operators such as `sparse_submanifold_conv3d` and `grid_sample_3d`. To optimize performance, you can tune two critical parameters:

- **`algorithm`**: Set via `flex_gemm.ops.spconv.set_algorithm()` or passed directly to convolution calls. Use `"auto"` to let the library select the best kernel for the current sparsity pattern, or specify `"spconv"` or `"grid_sample"` for deterministic behavior.
- **`hashmap_ratio`**: Configure with `flex_gemm.ops.spconv.set_hashmap_ratio()`. Higher values (e.g., 1.5) improve collision handling in sparse convolutions at the cost of increased memory consumption.

### Sparse Convolution Workflows

FlexGEMM handles the computational bottleneck in sparse 3D convolutions that process voxel features. Unlike dense operations that waste cycles on empty space, the library's kernels skip zero-valued voxels using the coordinate indices generated by the CUDA rasterization stage. This tight integration between the `o-voxel` CUDA backend and FlexGEMM frontend ensures that data stays on the GPU between voxelization and neural processing steps.

## Performance Optimization Strategies

### Kernel Launch Configuration

Tune the CUDA block size in `o-voxel/src/rasterize/rasterize.cu` to match your GPU's Streaming Multiprocessor (SM) count. For most NVIDIA architectures, blocks of 128-256 threads provide optimal occupancy. You can programmatically query the ideal configuration using `cudaOccupancyMaxPotentialBlockSize` before launching the `mesh_to_flexible_dual_grid` conversion.

### Memory Layout and Bandwidth

Store voxel coordinates in the packed 32-bit format produced by the Z-order and Hilbert curve utilities to reduce memory bandwidth by up to 50% compared to raw float32 coordinates. Ensure tensors align to 256-byte boundaries for optimal memory coalescing when transferring data between the voxel grid and FlexGEMM operations.

### FlexGEMM Algorithm Tuning

Experiment with the `FLEX_GEMM_ALGO` environment variable or runtime parameters in [`conv_flex_gemm.py`](https://github.com/microsoft/TRELLIS.2/blob/main/conv_flex_gemm.py). For highly sparse scenes (fewer than 1% occupancy), the `"spconv"` algorithm typically outperforms `"grid_sample"`, while the reverse holds for moderately dense volumes. Set `hashmap_ratio` between 1.2 and 2.0 depending on your scene complexity—higher ratios prevent hash collisions that stall the GPU during feature aggregation.

### Batch Processing

Group multiple meshes into a single batch before calling `o_voxel.convert.mesh_to_flexible_dual_grid()` to amortize kernel launch overhead. The rasterization kernels in `rasterize.cu` process batches more efficiently than sequential single-mesh conversions because they can maintain warps full of active threads across the entire batch.

## Implementation Examples

The following examples demonstrate the complete optimization workflow, from mesh voxelization through FlexGEMM-accelerated sparse convolution.

**Example 1: Basic Voxelization and Conversion Pipeline**

```python
import torch
import o_voxel
from flex_gemm.ops.spconv import sparse_submanifold_conv3d

# Load mesh data (verts: Nx3, faces: Mx3)

verts, faces = load_obj('example.obj')

# Step 1: Convert mesh to sparse voxel grid using optimized CUDA kernels

voxel_idx, dual_vert, intersect = o_voxel.convert.mesh_to_flexible_dual_grid(
    verts, faces, 
    voxel_size=0.01
)

# Step 2: Prepare features and weights for FlexGEMM processing

features = torch.randn(voxel_idx.shape[0], 64, device='cuda')
weight = torch.randn(64, 128, 3, 3, 3, device='cuda')

# Step 3: Apply sparse submanifold convolution

out = sparse_submanifold_conv3d(
    features, 
    voxel_idx, 
    weight,
    algorithm='auto',
    hashmap_ratio=1.5
)

# Step 4: Reconstruct mesh if needed

verts_rec, faces_rec = o_voxel.convert.flexible_dual_grid_to_mesh(
    voxel_idx, dual_vert, intersect
)

```

**Example 2: Tuning Rasterization with Custom Block Sizes**

```python
import torch
import o_voxel

# Configure GPU for deterministic behavior

torch.backends.cuda.matmul.allow_tf32 = False
torch.cuda.set_device(0)

# Load geometry

verts, faces = load_obj('complex_scene.obj')

# Override default kernel launch parameters from rasterize.cu

# Higher block sizes (256) improve throughput for large meshes

voxel_idx, dual_vert, intersect = o_voxel.convert.mesh_to_flexible_dual_grid(
    verts, 
    faces,
    voxel_size=0.005,
    block_size=256  # Matches optimized occupancy in rasterize.cu

)

```

**Example 3: End-to-End Optimized Pipeline**

```python
import torch
import o_voxel
import flex_gemm.ops.spconv as spconv

def render_optimized_mesh(mesh_path):
    verts, faces = load_obj(mesh_path)
    
    # Voxelize with optimal memory layout

    vox_idx, vox_vert, intersect = o_voxel.convert.mesh_to_flexible_dual_grid(
        verts, faces, 
        voxel_size=0.01
    )
    
    # Feature extraction with FlexGEMM

    feats = torch.randn(vox_idx.shape[0], 32, device='cuda')
    weight = torch.randn(32, 64, 3, 3, 3, device='cuda')
    
    conv_out = spconv.sparse_submanifold_conv3d(
        feats, 
        vox_idx, 
        weight,
        algorithm='auto',
        hashmap_ratio=1.2  # Balanced for medium-density scenes

    )
    
    # Reconstruct with dual grid conversion

    new_verts, new_faces = o_voxel.convert.flexible_dual_grid_to_mesh(
        vox_idx, vox_vert, intersect
    )
    
    return new_verts, new_faces, conv_out

```

## Summary

- **Custom CUDA kernels** in `o-voxel/src/rasterize/rasterize.cu` and `o-voxel/src/hash/hash.cu` handle geometric conversion and spatial indexing with warp-level optimization.
- **Space-filling curves** (`z_order.cu`, `hilbert.cu`) compact 3D coordinates to minimize memory bandwidth during grid operations.
- **FlexGEMM integration** through [`trellis2/modules/sparse/conv/conv_flex_gemm.py`](https://github.com/microsoft/TRELLIS.2/blob/main/trellis2/modules/sparse/conv/conv_flex_gemm.py) accelerates sparse convolutions via tunable algorithms and hashmap ratios.
- **Performance tuning** requires aligning block sizes to GPU architectures, batching mesh processing, and selecting FlexGEMM algorithms based on scene sparsity.

## Frequently Asked Questions

### How does TRELLIS.2 handle memory efficiency in sparse voxel grids?

TRELLIS.2 implements hash-based storage in `o-voxel/src/hash/hash.cu` combined with space-filling curve encodings from `z_order.cu` and `hilbert.cu`. These components pack 3D coordinates into 32-bit integers that consume less memory than float32 tuples while maintaining cache-friendly access patterns during spatial lookups.

### What is the optimal hashmap_ratio for FlexGEMM operations in TRELLIS.2?

The optimal `hashmap_ratio` depends on your scene density. For sparse scenes with scattered voxels, use ratios between 1.2 and 1.5 to prevent hash collisions. Denser volumes benefit from ratios up to 2.0, though this increases GPU memory consumption. Configure this parameter in [`conv_flex_gemm.py`](https://github.com/microsoft/TRELLIS.2/blob/main/conv_flex_gemm.py) or via `flex_gemm.ops.spconv.set_hashmap_ratio()`.

### Can I modify the CUDA kernel block sizes in the rasterization stage?

Yes. While the default configurations in `o-voxel/src/rasterize/rasterize.cu` target 128-256 threads for general GPU compatibility, you can override these at the Python API level using the `block_size` parameter in `mesh_to_flexible_dual_grid()`. Use `cudaOccupancyMaxPotentialBlockSize` profiling to determine the ideal configuration for your specific GPU architecture.

### How do I install FlexGEMM for TRELLIS.2 optimization?

Run the setup script with the `--flexgemm` flag to install FlexGEMM as a submodule and ensure CUDA toolchain compatibility. The installation process compiles the sparse convolution kernels against your system's CUDA version, enabling the optimized paths in [`trellis2/modules/sparse/conv/conv_flex_gemm.py`](https://github.com/microsoft/TRELLIS.2/blob/main/trellis2/modules/sparse/conv/conv_flex_gemm.py).