LingBot-Map Output Prediction Format and Post-Processing Pipeline Explained

LingBot-Map returns a dictionary of tensors containing camera pose encodings, depth maps, and 3D world points, which undergo a three-stage post-processing pipeline to convert pose encodings into extrinsic/intrinsic matrices, move data to CPU, and prepare visualization-ready NumPy arrays.

The Robbyant/lingbot-map repository implements a neural mapping system that processes video frames through a transformer-based architecture. Understanding the exact output format and how raw predictions are refined is critical for integrating the model into downstream 3D reconstruction workflows or custom visualization tools.

Raw Prediction Output Format

The base model defined in lingbot_map/models/gct_base.py produces a dictionary of tensors during the forward pass. The forward() method returns this dictionary when inference heads are enabled, containing batched tensors that encode geometric information for each frame.

Tensor Dictionary Structure

When the model processes an input batch of shape [B, S, 3, H, W] (batch, sequence, channels, height, width), the returned dictionary contains the following keys:

Key	Shape	Description
pose_enc	[B, S, 9]	Compact 9-vector camera pose encoding representing the world-to-camera transformation
depth	[B, S, H, W, 1]	Per-pixel depth estimates in meters
depth_conf	[B, S, H, W]	Confidence scores for each depth prediction
world_points	[B, S, H, W, 3]	3D point cloud coordinates in world space derived from depth
world_points_conf	[B, S, H, W]	Confidence values for each 3D point
images	[B, S, 3, H, W]	The pre-processed RGB frames fed to the model

Additional keys such as extrinsic, intrinsic, chunk_scales, and chunk_transforms are injected during subsequent post-processing stages. The dimensionality constants are captured in the _BATCHED_NDIMS mapping defined in demo.py.

Post-Processing Pipeline

The postprocess() function in demo.py transforms raw GPU tensors into clean, CPU-resident data structures suitable for analysis or visualization. This pipeline operates in three distinct stages.

Stage 1: Pose Decoding

The pipeline first converts the compact pose encoding into standard camera matrices. The function calls pose_encoding_to_extri_intri() from lingbot_map/utils/pose_enc.py to decode the [B, S, 9] pose_enc tensor into:

• extrinsic: A world-to-camera (w2c) 3×4 transformation matrix per frame
• intrinsic: Camera calibration parameters including focal length and principal point

Stage 2: Coordinate System Conversion

Since most visualization and rendering pipelines expect camera-to-world (c2w) coordinates, the pipeline converts the extrinsic matrices. The w2c matrix is padded to a 4×4 homogeneous matrix and inverted using closed_form_inverse_se3_general() from lingbot_map/utils/geometry.py. The resulting c2w matrix overwrites the predictions["extrinsic"] entry.

Stage 3: Cleanup and CPU Transfer

The final stage prepares the data for CPU consumption:

1. Dictionary Cleanup: The original pose_enc and temporary images entries are removed from the dictionary using pop() to reduce memory footprint
2. Device Transfer: All remaining tensors are moved to CPU memory via .to("cpu", non_blocking=True)
3. Batch Squeeze: The helper _squeeze_single_batch() checks _BATCHED_NDIMS to identify tensors where the first dimension equals 1, then squeezes them to remove the batch dimension
4. Synchronization: If CUDA is active, torch.cuda.synchronize() ensures all asynchronous GPU operations complete before the data is accessed

The function returns two objects: a cleaned predictions dictionary containing extrinsic, intrinsic, depth, depth_conf, world_points, and world_points_conf tensors, and images_cpu containing the original RGB frames.

Visualization-Ready Conversion

The prepare_for_visualization() function in demo.py performs the final conversion to NumPy arrays. This function calls _squeeze_single_batch() on every tensor, detaches them from the computation graph with .detach(), and converts them to NumPy arrays using .numpy().

The resulting vis_predictions dictionary contains arrays compatible with the visualization utilities in lingbot_map/vis/*, such as point_cloud_viewer.py and viser_wrapper.py. This format enables direct rendering of point clouds and camera frustums without additional tensor processing.

Practical Implementation

The following example demonstrates the complete workflow from raw model output to visualization-ready data:

import torch
from lingbot_map.models.gct_stream import GCTStream
from demo import postprocess, prepare_for_visualization

# Initialize model and dummy input

model = GCTStream()
images = torch.rand(1, 10, 3, 378, 518)  # [B, S, C, H, W]

# Stage 1: Forward pass produces raw prediction dictionary

raw_pred = model(images)  # Contains pose_enc, depth, world_points, etc.

# Stage 2: Post-processing decodes poses and moves to CPU

clean_pred, images_cpu = postprocess(raw_pred, images)

# Stage 3: Prepare for visualization

vis_dict = prepare_for_visualization(clean_pred, images=images_cpu)

# Access processed data

print(vis_dict["extrinsic"].shape)      # (S, 3, 4) camera-to-world matrices

print(vis_dict["depth"].shape)         # (S, H, W, 1) depth maps

print(vis_dict["world_points"].shape)   # (S, H, W, 3) point cloud

Summary

• LingBot-Map outputs a dictionary of batched tensors from lingbot_map/models/gct_base.py containing pose_enc, depth, world_points, and confidence maps
• Post-processing occurs in demo.py via three stages: pose decoding with pose_encoding_to_extri_intri(), coordinate inversion using closed_form_inverse_se3_general(), and CPU transfer with batch dimension squeezing
• The pipeline converts the raw 9-vector pose encoding into standard 3×4 extrinsic and intrinsic matrices, then transforms world-to-camera matrices into camera-to-world format
• Final output consists of NumPy arrays ready for consumption by visualization tools in the lingbot_map/vis package

Frequently Asked Questions

What is the shape of the pose_enc tensor in LingBot-Map?

The pose_enc tensor has shape [B, S, 9] where B is the batch size and S is the sequence length. This compact 9-vector encoding represents the camera pose in a compressed format that the post-processing pipeline expands into full 3×4 extrinsic and intrinsic matrices using pose_encoding_to_extri_intri().

How does LingBot-Map convert world-to-camera to camera-to-world coordinates?

The conversion happens in demo.py using closed_form_inverse_se3_general() from lingbot_map/utils/geometry.py. The function pads the 3×4 world-to-camera matrix to a 4×4 homogeneous matrix, computes the matrix inverse, and stores the result back into the extrinsic key as the camera-to-world transformation.

Why does the post-processing pipeline remove the batch dimension?

The _squeeze_single_batch() helper removes the batch dimension when it equals 1 to produce frame-sequence data that matches the expected input format for visualization tools. This conversion references the _BATCHED_NDIMS dictionary to identify which tensors contain batch dimensions and should be squeezed for downstream consumption.

What visualization tools consume the post-processed output?

The lingbot_map/vis package contains utilities such as point_cloud_viewer.py and viser_wrapper.py that expect the NumPy array format produced by prepare_for_visualization(). These tools render 3D point clouds, camera frustums, and depth maps using the processed extrinsic, world_points, and depth arrays.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}