# Understanding the Offline Rendering Pipeline for LingBot-Map Output

> Explore the offline rendering pipeline for LingBot-Map output. Learn how it converts GCT predictions to GLB files locally, featuring confidence filtering, sky segmentation, and scene alignment.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: deep-dive
- Published: 2026-07-06

---

**The offline rendering pipeline for LingBot-Map output converts raw GCT model predictions into self-contained GLB files through confidence-based filtering, optional sky segmentation, camera frustum generation, and scene alignment, all executing locally without network dependencies.**

The Robbyant/lingbot-map repository delivers a fully self-contained visualization system that transforms depth maps and point clouds into portable 3D assets. Understanding the offline rendering pipeline for LingBot-Map output enables researchers to generate publication-ready reconstructions without relying on cloud-based renderers or external visualization servers.

## Input Data Requirements

The pipeline entry point is a Python dictionary containing tensors produced by the GCT model architecture defined in [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py). This prediction dictionary must contain the following fields:

- **`world_points`** or **`world_points_from_depth`**: Tensor of shape `(S, H, W, 3)` representing 3D coordinates in the world frame for each of `S` frames
- **`world_points_conf`** or **`depth_conf`**: Tensor of shape `(S, H, W)` providing per-point confidence scores
- **`images`**: RGB frames with shape `(S, 3, H, W)` or `(S, H, W, 3)`
- **`extrinsic`**: Camera-to-world matrices of shape `(S, 3, 4)`
- **`intrinsic`**: Camera calibration matrices of shape `(S, 3, 3)` required for frustum geometry

These tensors typically originate from benchmark utilities that serialize model predictions to disk.

## Rendering Mode Selection

The `predictions_to_glb` function in [`lingbot_map/vis/glb_export.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/glb_export.py) (lines 35-46) supports two mutually exclusive reconstruction modes controlled by the `prediction_mode` parameter:

- **"Predicted Pointmap"**: Directly utilizes the pre-computed `world_points` field from the input dictionary
- **"Predicted Depthmap"**: Re-projects depth maps on-the-fly using `unproject_depth_map_to_point_map` to generate 3D coordinates dynamically

## Point Filtering and Masking Pipeline

### Sky Segmentation

When `mask_sky=True` and a source image directory is provided, the pipeline executes an ONNX-based sky segmentation model (`skyseg.onnx`). The `_apply_sky_mask` function (lines 115-144 in [`lingbot_map/vis/glb_export.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/glb_export.py)) downloads the model on first use, runs inference via `run_skyseg`, and multiplies the confidence map by the resulting binary sky mask to eliminate atmospheric points.

### Confidence Thresholding

The pipeline flattens the confidence tensor and applies a percentile-based threshold:

```python
conf = pred_world_points_conf.reshape(-1)
conf_threshold = np.percentile(conf, conf_thres) if conf_thres > 0 else 0.0
conf_mask = (conf >= conf_threshold) & (conf > 1e-5)

```

Points failing this threshold or possessing near-zero confidence values are discarded from the final geometry.

### Background Color Masking

Optional boolean flags enable removal of points matching **black** or **white** background colors (common in indoor RGB-D captures). These masks combine with `conf_mask` during the filtering stage (lines 142-152).

## Geometry Construction

### Point Cloud Generation

Surviving vertices and their associated RGB colors instantiate a `trimesh.PointCloud` object:

```python
scene_3d = trimesh.Scene()
point_cloud_data = trimesh.PointCloud(vertices=vertices_3d, colors=colors_rgb)
scene_3d.add_geometry(point_cloud_data)

```

This construction occurs in [`lingbot_map/vis/glb_export.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/glb_export.py) (lines 170-174).

### Camera Frustum Integration

Each camera extrinsic matrix converts to a 4×4 homogeneous transform and receives **OpenGL-compatible** formatting via `get_opengl_conversion_matrix()` (lines 336-342), which flips the Y and Z axes to match graphics pipeline conventions.

The `integrate_camera_into_scene` helper (lines 447-508) constructs cone-shaped frustum meshes, applies optional edge thickening via `frustum_thickness`, scales geometry relative to scene extent, and injects the result into the GLB scene graph.

## Scene Alignment and Export

### Coordinate System Alignment

The `apply_scene_alignment` function (lines 191-208) re-orients the entire scene using the first camera's extrinsic matrix. This transformation ensures the world-up axis points vertically and the forward direction faces toward the camera, providing consistent viewing orientation across different captures.

### Optional Trajectory Visualization

When enabled, `_build_trajectory_tube` (found in [`lingbot_map/vis/point_cloud_viewer.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/point_cloud_viewer.py), lines 996-1052) generates a continuous path connecting all camera positions. The implementation creates cylindrical segments between consecutive poses, coloring each section with a Matplotlib colormap gradient for temporal visualization.

### Final GLB Serialization

The `scene_3d.export(output_path)` method writes the binary GLB file. The output path defaults to `export.glb` but remains user-configurable through the UI's `self.glb_output_path` attribute.

## Implementation Examples

### Basic Offline Export

To generate a GLB file without launching the interactive viewer:

```python
from lingbot_map.vis.glb_export import predictions_to_glb

# `preds` contains the tensor dictionary described above

glb_scene = predictions_to_glb(
    predictions=preds,
    conf_thres=50.0,            # retain top 50% confident points

    mask_sky=True,              # requires image folder for segmentation

    target_dir="my_dataset/",   # directory containing images/ subdirectory

    prediction_mode="Predicted Pointmap",
)

glb_scene.export("my_output_scene.glb")
print("Offline GLB exported to my_output_scene.glb")

```

### Interactive Viewer with Export Options

The `PointCloudViewer` class in [`lingbot_map/vis/point_cloud_viewer.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/point_cloud_viewer.py) provides a local web interface for exploration and export:

```python
from lingbot_map.vis.point_cloud_viewer import PointCloudViewer

viewer = PointCloudViewer(
    pred_dict=preds,
    mask_sky=True,
    image_folder="my_dataset/",
    show_camera=True,
    glb_output_path="my_output_scene.glb",
)

viewer.animate()  # Launches local Viser web UI (offline-capable)

```

## Summary

- The **offline rendering pipeline for LingBot-Map output** operates entirely within [`lingbot_map/vis/glb_export.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/glb_export.py) and requires no network connectivity
- Input requires a structured prediction dictionary with world points, confidence scores, camera parameters, and optional RGB images
- **Two rendering modes** support either direct point cloud consumption or dynamic depth re-projection
- **Confidence filtering** uses percentile-based thresholding to remove low-quality reconstructions
- **Sky segmentation** via ONNX model (`skyseg.onnx`) automatically masks atmospheric regions when enabled
- **Camera frustums** convert to OpenGL-compatible coordinates and integrate as cone geometries with configurable thickness
- **Scene alignment** automatically orients the reconstruction using the first camera pose
- Output produces standard **GLB files** compatible with WebGL viewers, Blender, and three.js applications

## Frequently Asked Questions

### What input format does the offline rendering pipeline require?

The pipeline expects a Python dictionary containing PyTorch or NumPy tensors with specific keys: `world_points` (or `world_points_from_depth`), `world_points_conf` (or `depth_conf`), `images`, `extrinsic`, and `intrinsic`. These tensors typically originate from the GCT model implemented in [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py) and processed through benchmark utilities.

### How does the sky segmentation feature work without internet connectivity?

The pipeline downloads the `skyseg.onnx` model file on first use and caches it locally. Subsequent runs use the cached ONNX runtime entirely offline. The `_apply_sky_mask` function in [`lingbot_map/vis/glb_export.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/glb_export.py) handles inference using this local model, requiring only the original RGB image folder as input to generate binary sky masks.

### Can I customize the camera visualization in the exported GLB file?

Yes, the `integrate_camera_into_scene` function accepts a `frustum_thickness` parameter that controls the visual weight of camera frustum edges. Additionally, the `PointCloudViewer` class exposes UI controls for adjusting camera scaling, enabling trajectory tubes via `_build_trajectory_tube`, and configuring the output path through `glb_output_path`.

### Why does the pipeline use OpenGL coordinate conversion?

The `get_opengl_conversion_matrix()` function in [`lingbot_map/vis/glb_export.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/glb_export.py) (lines 336-342) flips the Y and Z axes to align the camera coordinate system with OpenGL conventions. This ensures that when the exported GLB file loads in standard WebGL viewers (such as three.js or Sketchfab), the camera frustums and point clouds appear with correct orientation and handedness without requiring additional transformation matrices.