# How LingBot-Map Performs Point Cloud Unprojection from Depth Predictions

> Learn how LingBot-Map unprojects point clouds from depth predictions using a three-step pipeline with camera intrinsics, extrinsic matrix inversion, and batch processing. Explore the code in geometry.py.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: deep-dive
- Published: 2026-07-04

---

**LingBot-Map converts depth predictions into 3D world-coordinate point clouds using a three-step pipeline involving camera intrinsics, extrinsic matrix inversion, and batch processing, implemented in [`geometry.py`](https://github.com/Robbyant/lingbot-map/blob/main/geometry.py).**

The process of point cloud unprojection from depth predictions transforms 2D distance maps into 3D spatial coordinates that robotic systems and visualizers can consume. In the LingBot-Map repository, this conversion operates through a deterministic geometric pipeline that maps pixel depths first to camera space, then to world space, using the camera's intrinsic and extrinsic parameters.

## The Three-Step Unprojection Pipeline

The pipeline implemented in [`lingbot_map/utils/geometry.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/geometry.py) processes depth predictions through sequential coordinate transformations. Each step handles a specific geometric operation required to move from image pixels to metric world coordinates.

### Step 1: Unproject Depth to Camera Space

The function `depth_to_cam_coords_points` (lines 121-125 in [`geometry.py`](https://github.com/Robbyant/lingbot-map/blob/main/geometry.py)) converts each pixel `(u, v)` with depth `d` into 3D camera coordinates using the intrinsic matrix `K`:

```python
K = [[fx,  0, cx],
     [ 0, fy, cy],
     [ 0,  0,  1]]

```

The mathematical implementation follows the standard pinhole camera model:

- `x_cam = (u - cx) * d / fx`
- `y_cam = (v - cy) * d / fy`
- `z_cam = d`

The routine generates a meshgrid of pixel coordinates `(u, v)`, applies these formulas element-wise, and stacks the results into an `(H, W, 3)` array representing the point cloud in camera space.

### Step 2: Transform Camera Points to World Space

Once points exist in camera coordinates, the `depth_to_world_coords_points` function (lines 90-92 in [`geometry.py`](https://github.com/Robbyant/lingbot-map/blob/main/geometry.py)) transforms them into world coordinates using the camera extrinsic matrix `E = [R | t]`.

The transformation requires inverting the extrinsic matrix to obtain the camera-to-world transform. The `closed_form_inverse_se3` helper (lines 84-88) computes this inverse efficiently:

- Rotation component: `R^T` (transpose of the rotation matrix)
- Translation component: `-R^T * t`

The final world coordinates are computed as:

```python
world_points = (cam_points @ R.T) + t

```

This operation rotates and translates the camera-relative points into the global coordinate frame used by mapping and visualization modules.

### Step 3: Batch Processing for Multiple Frames

The entry point `unproject_depth_map_to_point_map` (lines 25-54 in [`geometry.py`](https://github.com/Robbyant/lingbot-map/blob/main/geometry.py)) orchestrates the full pipeline across batched inputs. This function:

1. Accepts depth maps of shape `(S, H, W, 1)` where `S` is the batch size (number of frames)
2. Iterates over the batch dimension, calling the camera-space and world-space conversion functions for each frame
3. Stacks the results into a final tensor of shape `(S, H, W, 3)` containing XYZ world coordinates for every pixel in every frame

The implementation handles both NumPy arrays and PyTorch tensors, automatically moving tensor inputs to CPU and converting to NumPy (lines 39-44) before processing.

## Working with the Unprojection API

### NumPy Implementation Example

You can unproject depth predictions directly using the geometry utilities:

```python
import numpy as np
from lingbot_map.utils.geometry import unproject_depth_map_to_point_map

# Simulate a batch of 2 depth maps (480×640 resolution)

depth = np.random.rand(2, 480, 640, 1).astype(np.float32) * 5.0  # meters

# Identity camera poses (S, 3, 4)

extrinsics = np.tile(np.hstack([np.eye(3), np.zeros((3, 1))]), (2, 1, 1))

# Camera intrinsics: fx=fy=600, cx=320, cy=240 (S, 3, 3)

intrinsics = np.tile(np.array([[600, 0, 320],
                               [0, 600, 240],
                               [0,   0,   1]], dtype=np.float32), (2, 1, 1))

# Generate world point cloud (S, H, W, 3)

world_pts = unproject_depth_map_to_point_map(depth, extrinsics, intrinsics)
print(world_pts.shape)  # Output: (2, 480, 640, 3)

```

### Integration with Visualization Tools

The unprojection pipeline powers the interactive 3D viewers in the repository:

```python
from lingbot_map.vis.point_cloud_viewer import PointCloudViewer

# Assume depth_map, cam_extr, cam_int are tensors from model inference

viewer = PointCloudViewer()
viewer.update(depth_map, cam_extr, cam_int)  # Internally calls unproject_depth_map_to_point_map

viewer.show()

```

The `PointCloudViewer` class (line 31 in [`point_cloud_viewer.py`](https://github.com/Robbyant/lingbot-map/blob/main/point_cloud_viewer.py)) passes tensors directly to `unproject_depth_map_to_point_map`, then renders the resulting point cloud using OpenGL. Similarly, [`viser_wrapper.py`](https://github.com/Robbyant/lingbot-map/blob/main/viser_wrapper.py) (lines 76-84) utilizes the same function to prepare point clouds for the Viser visualization framework.

## Summary

- **Point cloud unprojection from depth predictions** requires three sequential transformations: pixel-to-camera via intrinsics, camera-to-world via inverted extrinsics, and batch aggregation.
- The `unproject_depth_map_to_point_map` function in [`lingbot_map/utils/geometry.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/geometry.py) serves as the primary API, returning tensors of shape `(S, H, W, 3)`.
- Camera calibration parameters (`fx`, `fy`, `cx`, `cy`) define the ray directions that lift 2D pixels into 3D camera space.
- The `closed_form_inverse_se3` utility efficiently computes the inverse SE(3) transform needed for coordinate system conversion.
- Visualization modules including [`point_cloud_viewer.py`](https://github.com/Robbyant/lingbot-map/blob/main/point_cloud_viewer.py) and [`viser_wrapper.py`](https://github.com/Robbyant/lingbot-map/blob/main/viser_wrapper.py) depend on this geometric pipeline to generate interactive 3D displays from raw depth predictions.

## Frequently Asked Questions

### What input shapes does `unproject_depth_map_to_point_map` expect?

The function requires three inputs: a depth map tensor of shape `(S, H, W, 1)` representing S frames of height H and width W, an extrinsic matrix of shape `(S, 3, 4)` containing the camera pose `[R | t]`, and an intrinsic matrix of shape `(S, 3, 3)` containing the calibration parameters. The output maintains the spatial dimensions while expanding the channel dimension to 3 for XYZ coordinates.

### How does LingBot-Map handle camera pose inversion?

The repository uses the `closed_form_inverse_se3` function (lines 84-88 in [`geometry.py`](https://github.com/Robbyant/lingbot-map/blob/main/geometry.py)) to analytically invert the extrinsic matrix. Rather than using numerical matrix inversion, it exploits the structure of SE(3) transforms: the inverse rotation is the transpose of `R`, and the inverse translation is `-R^T * t`. This approach is both numerically stable and computationally efficient for batched operations.

### Can the unprojection functions process PyTorch tensors?

Yes. The `unproject_depth_map_to_point_map` function automatically detects tensor inputs and moves them to CPU memory before converting to NumPy arrays (lines 39-44). This ensures compatibility with PyTorch training pipelines while maintaining the geometric calculations in NumPy for consistency with the visualization backends.

### Which files utilize the point cloud unprojection pipeline?

The primary consumers are [`lingbot_map/vis/point_cloud_viewer.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/point_cloud_viewer.py) (line 31) and [`lingbot_map/vis/viser_wrapper.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/viser_wrapper.py) (lines 76-84). Both modules import `unproject_depth_map_to_point_map` to convert depth predictions into world-coordinate point clouds for interactive 3D rendering and inspection.