How LingBot-Map Performs Point Cloud Unprojection from Depth Predictions

LingBot-Map converts depth predictions into 3D world-coordinate point clouds using a three-step pipeline involving camera intrinsics, extrinsic matrix inversion, and batch processing, implemented in geometry.py.

The process of point cloud unprojection from depth predictions transforms 2D distance maps into 3D spatial coordinates that robotic systems and visualizers can consume. In the LingBot-Map repository, this conversion operates through a deterministic geometric pipeline that maps pixel depths first to camera space, then to world space, using the camera's intrinsic and extrinsic parameters.

The Three-Step Unprojection Pipeline

The pipeline implemented in lingbot_map/utils/geometry.py processes depth predictions through sequential coordinate transformations. Each step handles a specific geometric operation required to move from image pixels to metric world coordinates.

Step 1: Unproject Depth to Camera Space

The function depth_to_cam_coords_points (lines 121-125 in geometry.py) converts each pixel (u, v) with depth d into 3D camera coordinates using the intrinsic matrix K:

K = [[fx,  0, cx],
     [ 0, fy, cy],
     [ 0,  0,  1]]

The mathematical implementation follows the standard pinhole camera model:

• x_cam = (u - cx) * d / fx
• y_cam = (v - cy) * d / fy
• z_cam = d

The routine generates a meshgrid of pixel coordinates (u, v), applies these formulas element-wise, and stacks the results into an (H, W, 3) array representing the point cloud in camera space.

Step 2: Transform Camera Points to World Space

Once points exist in camera coordinates, the depth_to_world_coords_points function (lines 90-92 in geometry.py) transforms them into world coordinates using the camera extrinsic matrix E = [R | t].

The transformation requires inverting the extrinsic matrix to obtain the camera-to-world transform. The closed_form_inverse_se3 helper (lines 84-88) computes this inverse efficiently:

• Rotation component: R^T (transpose of the rotation matrix)
• Translation component: -R^T * t

The final world coordinates are computed as:

world_points = (cam_points @ R.T) + t

This operation rotates and translates the camera-relative points into the global coordinate frame used by mapping and visualization modules.

Step 3: Batch Processing for Multiple Frames

The entry point unproject_depth_map_to_point_map (lines 25-54 in geometry.py) orchestrates the full pipeline across batched inputs. This function:

1. Accepts depth maps of shape (S, H, W, 1) where S is the batch size (number of frames)
2. Iterates over the batch dimension, calling the camera-space and world-space conversion functions for each frame
3. Stacks the results into a final tensor of shape (S, H, W, 3) containing XYZ world coordinates for every pixel in every frame

The implementation handles both NumPy arrays and PyTorch tensors, automatically moving tensor inputs to CPU and converting to NumPy (lines 39-44) before processing.

Working with the Unprojection API

NumPy Implementation Example

You can unproject depth predictions directly using the geometry utilities:

import numpy as np
from lingbot_map.utils.geometry import unproject_depth_map_to_point_map

# Simulate a batch of 2 depth maps (480×640 resolution)

depth = np.random.rand(2, 480, 640, 1).astype(np.float32) * 5.0  # meters

# Identity camera poses (S, 3, 4)

extrinsics = np.tile(np.hstack([np.eye(3), np.zeros((3, 1))]), (2, 1, 1))

# Camera intrinsics: fx=fy=600, cx=320, cy=240 (S, 3, 3)

intrinsics = np.tile(np.array([[600, 0, 320],
                               [0, 600, 240],
                               [0,   0,   1]], dtype=np.float32), (2, 1, 1))

# Generate world point cloud (S, H, W, 3)

world_pts = unproject_depth_map_to_point_map(depth, extrinsics, intrinsics)
print(world_pts.shape)  # Output: (2, 480, 640, 3)

Integration with Visualization Tools

The unprojection pipeline powers the interactive 3D viewers in the repository:

from lingbot_map.vis.point_cloud_viewer import PointCloudViewer

# Assume depth_map, cam_extr, cam_int are tensors from model inference

viewer = PointCloudViewer()
viewer.update(depth_map, cam_extr, cam_int)  # Internally calls unproject_depth_map_to_point_map

viewer.show()

The PointCloudViewer class (line 31 in point_cloud_viewer.py) passes tensors directly to unproject_depth_map_to_point_map, then renders the resulting point cloud using OpenGL. Similarly, viser_wrapper.py (lines 76-84) utilizes the same function to prepare point clouds for the Viser visualization framework.

Summary

• Point cloud unprojection from depth predictions requires three sequential transformations: pixel-to-camera via intrinsics, camera-to-world via inverted extrinsics, and batch aggregation.
• The unproject_depth_map_to_point_map function in lingbot_map/utils/geometry.py serves as the primary API, returning tensors of shape (S, H, W, 3).
• Camera calibration parameters (fx, fy, cx, cy) define the ray directions that lift 2D pixels into 3D camera space.
• The closed_form_inverse_se3 utility efficiently computes the inverse SE(3) transform needed for coordinate system conversion.
• Visualization modules including point_cloud_viewer.py and viser_wrapper.py depend on this geometric pipeline to generate interactive 3D displays from raw depth predictions.

Frequently Asked Questions

What input shapes does unproject_depth_map_to_point_map expect?

The function requires three inputs: a depth map tensor of shape (S, H, W, 1) representing S frames of height H and width W, an extrinsic matrix of shape (S, 3, 4) containing the camera pose [R | t], and an intrinsic matrix of shape (S, 3, 3) containing the calibration parameters. The output maintains the spatial dimensions while expanding the channel dimension to 3 for XYZ coordinates.

How does LingBot-Map handle camera pose inversion?

The repository uses the closed_form_inverse_se3 function (lines 84-88 in geometry.py) to analytically invert the extrinsic matrix. Rather than using numerical matrix inversion, it exploits the structure of SE(3) transforms: the inverse rotation is the transpose of R, and the inverse translation is -R^T * t. This approach is both numerically stable and computationally efficient for batched operations.

Can the unprojection functions process PyTorch tensors?

Yes. The unproject_depth_map_to_point_map function automatically detects tensor inputs and moves them to CPU memory before converting to NumPy arrays (lines 39-44). This ensures compatibility with PyTorch training pipelines while maintaining the geometric calculations in NumPy for consistency with the visualization backends.

Which files utilize the point cloud unprojection pipeline?

The primary consumers are lingbot_map/vis/point_cloud_viewer.py (line 31) and lingbot_map/vis/viser_wrapper.py (lines 76-84). Both modules import unproject_depth_map_to_point_map to convert depth predictions into world-coordinate point clouds for interactive 3D rendering and inspection.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}