# LingBot-Map Output Prediction Format and Post-Processing Pipeline Explained

> Understand LingBot-Map's output prediction format and its three-stage post-processing pipeline. Learn how it converts tensors to matrices, moves data to CPU, and prepares NumPy arrays for visualization.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: explanation
- Published: 2026-07-01

---

**LingBot-Map returns a dictionary of tensors containing camera pose encodings, depth maps, and 3D world points, which undergo a three-stage post-processing pipeline to convert pose encodings into extrinsic/intrinsic matrices, move data to CPU, and prepare visualization-ready NumPy arrays.**

The `Robbyant/lingbot-map` repository implements a neural mapping system that processes video frames through a transformer-based architecture. Understanding the exact output format and how raw predictions are refined is critical for integrating the model into downstream 3D reconstruction workflows or custom visualization tools.

## Raw Prediction Output Format

The base model defined in [`lingbot_map/models/gct_base.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_base.py) produces a **dictionary of tensors** during the forward pass. The `forward()` method returns this dictionary when inference heads are enabled, containing batched tensors that encode geometric information for each frame.

### Tensor Dictionary Structure

When the model processes an input batch of shape `[B, S, 3, H, W]` (batch, sequence, channels, height, width), the returned dictionary contains the following keys:

| Key | Shape | Description |
|-----|-------|-------------|
| `pose_enc` | `[B, S, 9]` | Compact 9-vector camera pose encoding representing the world-to-camera transformation |
| `depth` | `[B, S, H, W, 1]` | Per-pixel depth estimates in meters |
| `depth_conf` | `[B, S, H, W]` | Confidence scores for each depth prediction |
| `world_points` | `[B, S, H, W, 3]` | 3D point cloud coordinates in world space derived from depth |
| `world_points_conf` | `[B, S, H, W]` | Confidence values for each 3D point |
| `images` | `[B, S, 3, H, W]` | The pre-processed RGB frames fed to the model |

Additional keys such as `extrinsic`, `intrinsic`, `chunk_scales`, and `chunk_transforms` are injected during subsequent post-processing stages. The dimensionality constants are captured in the `_BATCHED_NDIMS` mapping defined in [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py).

## Post-Processing Pipeline

The `postprocess()` function in [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py) transforms raw GPU tensors into clean, CPU-resident data structures suitable for analysis or visualization. This pipeline operates in three distinct stages.

### Stage 1: Pose Decoding

The pipeline first converts the compact pose encoding into standard camera matrices. The function calls `pose_encoding_to_extri_intri()` from [`lingbot_map/utils/pose_enc.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/pose_enc.py) to decode the `[B, S, 9]` `pose_enc` tensor into:

- `extrinsic`: A world-to-camera (w2c) 3×4 transformation matrix per frame
- `intrinsic`: Camera calibration parameters including focal length and principal point

### Stage 2: Coordinate System Conversion

Since most visualization and rendering pipelines expect camera-to-world (c2w) coordinates, the pipeline converts the extrinsic matrices. The w2c matrix is padded to a 4×4 homogeneous matrix and inverted using `closed_form_inverse_se3_general()` from [`lingbot_map/utils/geometry.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/geometry.py). The resulting c2w matrix overwrites the `predictions["extrinsic"]` entry.

### Stage 3: Cleanup and CPU Transfer

The final stage prepares the data for CPU consumption:

1. **Dictionary Cleanup**: The original `pose_enc` and temporary `images` entries are removed from the dictionary using `pop()` to reduce memory footprint
2. **Device Transfer**: All remaining tensors are moved to CPU memory via `.to("cpu", non_blocking=True)`
3. **Batch Squeeze**: The helper `_squeeze_single_batch()` checks `_BATCHED_NDIMS` to identify tensors where the first dimension equals 1, then squeezes them to remove the batch dimension
4. **Synchronization**: If CUDA is active, `torch.cuda.synchronize()` ensures all asynchronous GPU operations complete before the data is accessed

The function returns two objects: a cleaned `predictions` dictionary containing `extrinsic`, `intrinsic`, `depth`, `depth_conf`, `world_points`, and `world_points_conf` tensors, and `images_cpu` containing the original RGB frames.

## Visualization-Ready Conversion

The `prepare_for_visualization()` function in [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py) performs the final conversion to NumPy arrays. This function calls `_squeeze_single_batch()` on every tensor, detaches them from the computation graph with `.detach()`, and converts them to NumPy arrays using `.numpy()`.

The resulting `vis_predictions` dictionary contains arrays compatible with the visualization utilities in `lingbot_map/vis/*`, such as [`point_cloud_viewer.py`](https://github.com/Robbyant/lingbot-map/blob/main/point_cloud_viewer.py) and [`viser_wrapper.py`](https://github.com/Robbyant/lingbot-map/blob/main/viser_wrapper.py). This format enables direct rendering of point clouds and camera frustums without additional tensor processing.

## Practical Implementation

The following example demonstrates the complete workflow from raw model output to visualization-ready data:

```python
import torch
from lingbot_map.models.gct_stream import GCTStream
from demo import postprocess, prepare_for_visualization

# Initialize model and dummy input

model = GCTStream()
images = torch.rand(1, 10, 3, 378, 518)  # [B, S, C, H, W]

# Stage 1: Forward pass produces raw prediction dictionary

raw_pred = model(images)  # Contains pose_enc, depth, world_points, etc.

# Stage 2: Post-processing decodes poses and moves to CPU

clean_pred, images_cpu = postprocess(raw_pred, images)

# Stage 3: Prepare for visualization

vis_dict = prepare_for_visualization(clean_pred, images=images_cpu)

# Access processed data

print(vis_dict["extrinsic"].shape)      # (S, 3, 4) camera-to-world matrices

print(vis_dict["depth"].shape)         # (S, H, W, 1) depth maps

print(vis_dict["world_points"].shape)   # (S, H, W, 3) point cloud

```

## Summary

- **LingBot-Map outputs** a dictionary of batched tensors from [`lingbot_map/models/gct_base.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_base.py) containing `pose_enc`, `depth`, `world_points`, and confidence maps
- **Post-processing** occurs in [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py) via three stages: pose decoding with `pose_encoding_to_extri_intri()`, coordinate inversion using `closed_form_inverse_se3_general()`, and CPU transfer with batch dimension squeezing
- **The pipeline** converts the raw 9-vector pose encoding into standard 3×4 extrinsic and intrinsic matrices, then transforms world-to-camera matrices into camera-to-world format
- **Final output** consists of NumPy arrays ready for consumption by visualization tools in the `lingbot_map/vis` package

## Frequently Asked Questions

### What is the shape of the pose_enc tensor in LingBot-Map?

The `pose_enc` tensor has shape `[B, S, 9]` where B is the batch size and S is the sequence length. This compact 9-vector encoding represents the camera pose in a compressed format that the post-processing pipeline expands into full 3×4 extrinsic and intrinsic matrices using `pose_encoding_to_extri_intri()`.

### How does LingBot-Map convert world-to-camera to camera-to-world coordinates?

The conversion happens in [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py) using `closed_form_inverse_se3_general()` from [`lingbot_map/utils/geometry.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/geometry.py). The function pads the 3×4 world-to-camera matrix to a 4×4 homogeneous matrix, computes the matrix inverse, and stores the result back into the `extrinsic` key as the camera-to-world transformation.

### Why does the post-processing pipeline remove the batch dimension?

The `_squeeze_single_batch()` helper removes the batch dimension when it equals 1 to produce frame-sequence data that matches the expected input format for visualization tools. This conversion references the `_BATCHED_NDIMS` dictionary to identify which tensors contain batch dimensions and should be squeezed for downstream consumption.

### What visualization tools consume the post-processed output?

The `lingbot_map/vis` package contains utilities such as [`point_cloud_viewer.py`](https://github.com/Robbyant/lingbot-map/blob/main/point_cloud_viewer.py) and [`viser_wrapper.py`](https://github.com/Robbyant/lingbot-map/blob/main/viser_wrapper.py) that expect the NumPy array format produced by `prepare_for_visualization()`. These tools render 3D point clouds, camera frustums, and depth maps using the processed `extrinsic`, `world_points`, and `depth` arrays.