Understanding the Offline Rendering Pipeline for LingBot-Map Output

The offline rendering pipeline for LingBot-Map output converts raw GCT model predictions into self-contained GLB files through confidence-based filtering, optional sky segmentation, camera frustum generation, and scene alignment, all executing locally without network dependencies.

The Robbyant/lingbot-map repository delivers a fully self-contained visualization system that transforms depth maps and point clouds into portable 3D assets. Understanding the offline rendering pipeline for LingBot-Map output enables researchers to generate publication-ready reconstructions without relying on cloud-based renderers or external visualization servers.

Input Data Requirements

The pipeline entry point is a Python dictionary containing tensors produced by the GCT model architecture defined in lingbot_map/models/gct_stream.py. This prediction dictionary must contain the following fields:

• world_points or world_points_from_depth: Tensor of shape (S, H, W, 3) representing 3D coordinates in the world frame for each of S frames
• world_points_conf or depth_conf: Tensor of shape (S, H, W) providing per-point confidence scores
• images: RGB frames with shape (S, 3, H, W) or (S, H, W, 3)
• extrinsic: Camera-to-world matrices of shape (S, 3, 4)
• intrinsic: Camera calibration matrices of shape (S, 3, 3) required for frustum geometry

These tensors typically originate from benchmark utilities that serialize model predictions to disk.

Rendering Mode Selection

The predictions_to_glb function in lingbot_map/vis/glb_export.py (lines 35-46) supports two mutually exclusive reconstruction modes controlled by the prediction_mode parameter:

• "Predicted Pointmap": Directly utilizes the pre-computed world_points field from the input dictionary
• "Predicted Depthmap": Re-projects depth maps on-the-fly using unproject_depth_map_to_point_map to generate 3D coordinates dynamically

Point Filtering and Masking Pipeline

Sky Segmentation

When mask_sky=True and a source image directory is provided, the pipeline executes an ONNX-based sky segmentation model (skyseg.onnx). The _apply_sky_mask function (lines 115-144 in lingbot_map/vis/glb_export.py) downloads the model on first use, runs inference via run_skyseg, and multiplies the confidence map by the resulting binary sky mask to eliminate atmospheric points.

Confidence Thresholding

The pipeline flattens the confidence tensor and applies a percentile-based threshold:

conf = pred_world_points_conf.reshape(-1)
conf_threshold = np.percentile(conf, conf_thres) if conf_thres > 0 else 0.0
conf_mask = (conf >= conf_threshold) & (conf > 1e-5)

Points failing this threshold or possessing near-zero confidence values are discarded from the final geometry.

Background Color Masking

Optional boolean flags enable removal of points matching black or white background colors (common in indoor RGB-D captures). These masks combine with conf_mask during the filtering stage (lines 142-152).

Geometry Construction

Point Cloud Generation

Surviving vertices and their associated RGB colors instantiate a trimesh.PointCloud object:

scene_3d = trimesh.Scene()
point_cloud_data = trimesh.PointCloud(vertices=vertices_3d, colors=colors_rgb)
scene_3d.add_geometry(point_cloud_data)

This construction occurs in lingbot_map/vis/glb_export.py (lines 170-174).

Camera Frustum Integration

Each camera extrinsic matrix converts to a 4×4 homogeneous transform and receives OpenGL-compatible formatting via get_opengl_conversion_matrix() (lines 336-342), which flips the Y and Z axes to match graphics pipeline conventions.

The integrate_camera_into_scene helper (lines 447-508) constructs cone-shaped frustum meshes, applies optional edge thickening via frustum_thickness, scales geometry relative to scene extent, and injects the result into the GLB scene graph.

Scene Alignment and Export

Coordinate System Alignment

The apply_scene_alignment function (lines 191-208) re-orients the entire scene using the first camera's extrinsic matrix. This transformation ensures the world-up axis points vertically and the forward direction faces toward the camera, providing consistent viewing orientation across different captures.

Optional Trajectory Visualization

When enabled, _build_trajectory_tube (found in lingbot_map/vis/point_cloud_viewer.py, lines 996-1052) generates a continuous path connecting all camera positions. The implementation creates cylindrical segments between consecutive poses, coloring each section with a Matplotlib colormap gradient for temporal visualization.

Final GLB Serialization

The scene_3d.export(output_path) method writes the binary GLB file. The output path defaults to export.glb but remains user-configurable through the UI's self.glb_output_path attribute.

Implementation Examples

Basic Offline Export

To generate a GLB file without launching the interactive viewer:

from lingbot_map.vis.glb_export import predictions_to_glb

# `preds` contains the tensor dictionary described above

glb_scene = predictions_to_glb(
    predictions=preds,
    conf_thres=50.0,            # retain top 50% confident points

    mask_sky=True,              # requires image folder for segmentation

    target_dir="my_dataset/",   # directory containing images/ subdirectory

    prediction_mode="Predicted Pointmap",
)

glb_scene.export("my_output_scene.glb")
print("Offline GLB exported to my_output_scene.glb")

Interactive Viewer with Export Options

The PointCloudViewer class in lingbot_map/vis/point_cloud_viewer.py provides a local web interface for exploration and export:

from lingbot_map.vis.point_cloud_viewer import PointCloudViewer

viewer = PointCloudViewer(
    pred_dict=preds,
    mask_sky=True,
    image_folder="my_dataset/",
    show_camera=True,
    glb_output_path="my_output_scene.glb",
)

viewer.animate()  # Launches local Viser web UI (offline-capable)

Summary

• The offline rendering pipeline for LingBot-Map output operates entirely within lingbot_map/vis/glb_export.py and requires no network connectivity
• Input requires a structured prediction dictionary with world points, confidence scores, camera parameters, and optional RGB images
• Two rendering modes support either direct point cloud consumption or dynamic depth re-projection
• Confidence filtering uses percentile-based thresholding to remove low-quality reconstructions
• Sky segmentation via ONNX model (skyseg.onnx) automatically masks atmospheric regions when enabled
• Camera frustums convert to OpenGL-compatible coordinates and integrate as cone geometries with configurable thickness
• Scene alignment automatically orients the reconstruction using the first camera pose
• Output produces standard GLB files compatible with WebGL viewers, Blender, and three.js applications

Frequently Asked Questions

What input format does the offline rendering pipeline require?

The pipeline expects a Python dictionary containing PyTorch or NumPy tensors with specific keys: world_points (or world_points_from_depth), world_points_conf (or depth_conf), images, extrinsic, and intrinsic. These tensors typically originate from the GCT model implemented in lingbot_map/models/gct_stream.py and processed through benchmark utilities.

How does the sky segmentation feature work without internet connectivity?

The pipeline downloads the skyseg.onnx model file on first use and caches it locally. Subsequent runs use the cached ONNX runtime entirely offline. The _apply_sky_mask function in lingbot_map/vis/glb_export.py handles inference using this local model, requiring only the original RGB image folder as input to generate binary sky masks.

Can I customize the camera visualization in the exported GLB file?

Yes, the integrate_camera_into_scene function accepts a frustum_thickness parameter that controls the visual weight of camera frustum edges. Additionally, the PointCloudViewer class exposes UI controls for adjusting camera scaling, enabling trajectory tubes via _build_trajectory_tube, and configuring the output path through glb_output_path.

Why does the pipeline use OpenGL coordinate conversion?

The get_opengl_conversion_matrix() function in lingbot_map/vis/glb_export.py (lines 336-342) flips the Y and Z axes to align the camera coordinate system with OpenGL conventions. This ensures that when the exported GLB file loads in standard WebGL viewers (such as three.js or Sketchfab), the camera frustums and point clouds appear with correct orientation and handedness without requiring additional transformation matrices.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}