How a Point Cloud is Generated from LingBot-Map's Reconstruction: A Technical Deep Dive

LingBot-Map generates dense, colored point clouds in world coordinates by either loading pre-computed per-frame clouds or back-projecting depth maps using camera intrinsics and trajectory poses, with the core logic implemented in the BenchmarkLoader.load_point_cloud_grid() method.

The Robbyant/lingbot-map repository provides a complete pipeline for reconstructing 3D scenes from RGB-D sequences. Understanding how a point cloud is generated from LingBot-Map's reconstruction requires examining the benchmark/benchmark/core/loader.py module, where the transformation from raw sensor data to world-aligned 3D points occurs.

Core Generation Pipeline

The point cloud generation follows a deterministic pipeline that processes auxiliary data (RGB frames, camera intrinsics, and trajectory poses) to produce a dense (N, H, W, 6) tensor containing XYZ coordinates and RGB color values.

Loading Auxiliary Data

The process begins by loading the essential calibration and trajectory information. According to the source code in benchmark/benchmark/core/loader.py (lines 352-363 and 374-382), the loader initializes:

• rgb_list: The color frames for each timestamp
• intrinsics_array: Camera calibration parameters (fx, fy, cx, cy)
• traj_array: Camera-to-world poses (c2w) defining the trajectory

These arrays provide the geometric context required to transform 2D pixel measurements into 3D world coordinates.

Selecting the Data Source

At line 332 of loader.py, the system determines whether to use pre-computed point clouds or generate them from depth maps. The check use_point_cloud = (point_list is not None and all(p is not None for p in point_list)) enables dual-path execution:

1. Direct path: Use existing per-frame XYZ grids if available
2. Back-projection path: Compute 3D points from depth maps using camera geometry

Direct Point Cloud Path

When pre-computed points exist, the loader copies the stored XYZ and RGB data directly into the output tensor. As implemented in lines 77-82 of loader.py:

xyzrgb[i, :, :, :3] = xyz_grid
xyzrgb[i, :, :, 3:] = rgb.astype(np.float32) / 255.0

This path bypasses geometric computation, using the previously reconstructed geometry stored in the dataset.

Depth Back-Projection Path

If no pre-computed points exist, the system generates points via depth back-projection (lines 84-106). This process involves:

1. Generating pixel grids: Creating meshgrid coordinates uu, vv for all image pixels
2. Camera-space conversion: Computing 3D coordinates using the pinhole camera model with intrinsics
3. World-space transformation: Applying the c2w (camera-to-world) pose matrix to transform points into global coordinates

The mathematical operation follows the standard projection equations where depth values Z are converted to camera coordinates using X = (u - cx) * Z / fx and Y = (v - cy) * Z / fy, then transformed by the 4x4 pose matrix.

Masking and Filtering

Valid points are identified using a boolean mask created at lines 112-113, which checks for depth > 1e-4 and finite values. Optional filtering occurs at lines 124-148:

• Confidence thresholding: Discarding points below a global percentile threshold
• AOI masking: Applying region-of-interest masks from the mask/ directory

The final output at lines 149-150 returns the xyzrgb tensor with shape (N_frames, H, W, 6) and the corresponding validity mask.

Implementation Details in loader.py

The load_point_cloud_grid Method Signature

The primary entry point load_point_cloud_grid() (lines 401-493 in benchmark/benchmark/core/loader.py) accepts parameters controlling the generation behavior:

def load_point_cloud_grid(
    self,
    confidence_threshold: Optional[float] = None,
    use_aoi_mask: bool = False,
    remove_sky: bool = False
) -> Tuple[np.ndarray, np.ndarray]:

This method orchestrates the entire pipeline, delegating to helper methods for data loading and applying post-processing filters based on user configuration.

Coordinate Systems and Output Format

The resulting xyzrgb array contains:

• Spatial coordinates: X, Y, Z in meters (world coordinate system)
• Color values: R, G, B normalized to [0, 1] range

The mask tensor indicates which pixels contain valid 3D measurements, allowing downstream consumers to ignore invalid or filtered points.

Visualization and Export

Interactive Viewing with PointCloudViewer

The lingbot_map/vis/point_cloud_viewer.py module consumes the generated tensors to render interactive 3D scenes. The viewer's _add_point_cloud method (lines 1089-1112) creates visualization handles from the xyzrgb data.

When users trigger the "Regenerate all point clouds with current settings" command (lines 572-594), the viewer internally calls loader.load_point_cloud_grid() to respect updated confidence thresholds or segmentation options.

Exporting to GLB and PLY Formats

For external use, point clouds can be serialized using:

• PLY format: Via benchmark/benchmark/io/pointcloud.py for raw point storage
• GLB format: Via lingbot_map/vis/glb_export.py (lines 65-78), which constructs a trimesh.Scene combining point clouds and camera poses

Code Examples

Extracting Point Clouds via Python API

from benchmark.benchmark.core.loader import BenchmarkLoader

# Initialize loader with dataset artifact

loader = BenchmarkLoader(artifact)

# Generate point cloud with 50% confidence filtering

xyzrgb, mask = loader.load_point_cloud_grid(
    confidence_threshold=0.5,
    use_aoi_mask=False
)

# xyzrgb.shape: (N_frames, H, W, 6) - XYZRGB format

# mask.shape: (N_frames, H, W) - Boolean validity mask

Source reference: load_point_cloud_grid implementation in benchmark/benchmark/core/loader.py (lines 401-493).

Visualizing with the Built-in Viewer

from lingbot_map.vis.point_cloud_viewer import PointCloudViewer
from benchmark.benchmark.core.loader import BenchmarkLoader

loader = BenchmarkLoader(artifact)
viewer = PointCloudViewer(loader)

# Display interactive window

viewer.show()
viewer.toggle_global(True)  # Merge all frames into single cloud

Exporting to GLB

from lingbot_map.vis.glb_export import export_glb
from benchmark.benchmark.core.loader import BenchmarkLoader

loader = BenchmarkLoader(artifact)
xyzrgb, mask = loader.load_point_cloud_grid()

# Export to GLB format with camera poses

export_glb(xyzrgb, mask, out_path="reconstruction.glb")

Implementation details: Export logic in lingbot_map/vis/glb_export.py (lines 65-78).

Summary

• LingBot-Map generates point clouds through BenchmarkLoader.load_point_cloud_grid() in benchmark/benchmark/core/loader.py, supporting both pre-computed clouds and real-time depth back-projection.
• The dual-path architecture allows efficient reuse of cached reconstructions or fresh generation from depth maps using camera intrinsics and trajectory poses.
• World coordinates are produced by transforming camera-space points using the c2w (camera-to-world) pose matrices from the trajectory.
• Output format is a standardized (N, H, W, 6) float32 tensor containing XYZ coordinates in meters and RGB colors normalized to [0, 1].
• Post-processing includes confidence thresholding, AOI masking, and sky removal to improve point cloud quality.
• Visualization and export are handled by point_cloud_viewer.py and glb_export.py, supporting interactive inspection and standard 3D file formats.

Frequently Asked Questions

What coordinate system does LingBot-Map use for point clouds?

LingBot-Map generates point clouds in world coordinates (meters). The system transforms camera-space points using the camera-to-world (c2w) pose matrices stored in the trajectory array. This ensures that points from different frames align correctly in a unified global coordinate system, regardless of camera movement during capture.

How does LingBot-Map handle missing or low-confidence depth data?

The system creates a boolean validity mask (lines 112-113 in loader.py) that identifies valid points based on depth values greater than 1e-4 and finite checks. Additionally, an optional confidence threshold parameter filters out points below a specified global percentile (lines 124-148), allowing users to exclude noisy or uncertain measurements from the final point cloud.

Can I use pre-computed point clouds instead of generating them from depth?

Yes. If the dataset contains pre-computed per-frame point clouds (checked at line 332 via load_points_list()), the loader bypasses depth back-projection and copies the stored XYZ/RGB data directly into the output tensor (lines 77-82). This significantly accelerates loading when high-quality reconstructions are already available.

What file formats are supported for exporting point clouds?

The repository supports PLY format via benchmark/benchmark/io/pointcloud.py for raw point cloud storage, and GLB format via lingbot_map/vis/glb_export.py (lines 65-78) for combined geometry and camera pose export. The GLB exporter uses trimesh.Scene to create portable 3D assets suitable for visualization in external applications.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}