Prediction Heads in LingBot-Map: Camera Extrinsics and Dense Per-Pixel Outputs

LingBot-Map uses two specialized prediction heads—CameraHead for camera pose estimation and DPTHead for dense depth and normal prediction—to convert transformer token streams into concrete spatial outputs.

LingBot-Map, available in the Robbyant/lingbot-map repository, implements a modular "head" architecture that bridges the gap between transformer backbones and task-specific outputs. These prediction heads transform aggregated token representations into either camera extrinsics for SLAM-style odometry or dense per-pixel maps for scene understanding.

CameraHead and CameraCausalHead

The CameraHead family, defined in lingbot_map/heads/camera_head.py, handles camera extrinsics prediction, outputting translation vectors, quaternion rotations, and focal length or field-of-view parameters.

Architecture and Design

The CameraHead class processes a dedicated camera token through a small transformer "trunk" with configurable depth. The design employs an empty pose token as the initial input, iteratively refining the pose encoding across multiple iterations. Key implementation details include:

• Adaptive layer normalization: Token features are modulated with learnable shift, scale, and gate parameters.
• Flexible activation: Final outputs pass through activate_pose from lingbot_map/heads/head_act.py, supporting linear, ReLU, inverse-log, and other activation schemes.
• Multi-iteration refinement: The head can run multiple refinement steps (e.g., num_iterations=4) to progressively improve pose estimates.

Streaming Inference with CameraCausalHead

For video sequences, CameraCausalHead extends the base architecture with KV-cache support, enabling streaming inference without recomputing past attention. As implemented in the same file, this variant maintains a sliding-window cache (kv_cache) and supports optional 3-D RoPE positional encodings. This allows the model to process frames sequentially, making it suitable for real-time SLAM applications.

DPTHead for Dense Per-Pixel Prediction

The DPTHead class in lingbot_map/heads/dpt_head.py generates dense predictions such as depth maps, surface normals, or confidence maps from multi-scale transformer features.

Multi-Scale Feature Fusion

The head operates on token layers extracted at different depths of the backbone (configurable via intermediate_layer_idx). The pipeline:

1. Projects token maps to 2-D feature maps using learned projections.
2. Applies optional positional embeddings via utilities in lingbot_map/heads/utils.py (functions like create_uv_grid and position_grid_to_embed).
3. Up-samples features through learned transpose convolutions.
4. Fuses four spatial scales using FeatureFusionBlock modules in a scratch network configuration.

Final activations are handled by activate_head in head_act.py, with support for norm_exp, exp, relu, and inv_log modes depending on whether the output represents depth or confidence.

Activation Utilities

Both heads rely on shared activation functions defined in lingbot_map/heads/head_act.py:

• activate_pose: Applies component-specific activation to translation, quaternion, and focal length parameters.
• activate_head: Converts raw network outputs into 3-D points and confidence values with configurable activation schemes.

These utilities enable rapid experimentation with different output parameterizations without modifying the head implementations.

Implementation Examples

Camera Pose Prediction

import torch
from lingbot_map.heads.camera_head import CameraHead

# Simulated aggregated token list (output of the backbone)

# Shape: [B, N, C] where N includes a dedicated camera token at index 0

tokens = torch.randn(2, 128, 2048)          # B=2, N=128, C=2048

aggregated_tokens = [tokens]                # list of token tensors from each stage

camera_head = CameraHead(
    dim_in=2048,
    trunk_depth=4,
    pose_encoding_type="absT_quaR_FoV",
    num_heads=16,
    mlp_ratio=4,
    trans_act="linear",
    quat_act="linear",
    fl_act="relu",
)

# Predict camera pose with four refinement iterations

pose_predictions = camera_head(aggregated_tokens, num_iterations=4)

# `pose_predictions` is a list of length 4 where each entry has shape

# (B, 1, 9) → [translation(3), quaternion(4), focal‑len/FoV(2)]

print(pose_predictions[0].shape)   # torch.Size([2, 1, 9])

Dense Depth Prediction

import torch
from lingbot_map.heads.dpt_head import DPTHead

# Assume we have 4 token tensors from different transformer layers,

# each with shape (B, S, C) where S includes patch tokens after the camera token.

tokens_layer0 = torch.randn(2, 64, 2048)
tokens_layer1 = torch.randn(2, 64, 2048)
tokens_layer2 = torch.randn(2, 64, 2048)
tokens_layer3 = torch.randn(2, 64, 2048)

aggregated_tokens = [tokens_layer0, tokens_layer1, tokens_layer2, tokens_layer3]

# Dummy image batch (required only for spatial dimensions)

images = torch.randn(2, 1, 3, 224, 224)   # B, S, C, H, W

# Index where patch tokens start (camera token occupies index 0)

patch_start_idx = 1

dpt_head = DPTHead(
    dim_in=2048,
    patch_size=14,
    output_dim=1,             # 1‑channel depth map

    activation="norm_exp",
    conf_activation="expp1",
    features=256,
    out_channels=[256, 512, 1024, 1024],
    intermediate_layer_idx=[0, 1, 2, 3],
)

depth, confidence = dpt_head(aggregated_tokens, images, patch_start_idx)
print(depth.shape)        # torch.Size([2, 1, 1, 224, 224])

print(confidence.shape)   # torch.Size([2, 1, 1, 224, 224])

Streaming Camera Inference

from lingbot_map.heads.camera_head import CameraCausalHead

causal_head = CameraCausalHead(
    dim_in=2048,
    trunk_depth=4,
    num_iterations=3,
    enable_3d_rope=False,
    sliding_window_size=64,
)

# Process frames one by one (or in small batches)

for frame_idx in range(num_frames):
    pose_list = causal_head(aggregated_tokens, causal_inference=True)
    # pose_list contains refined poses for the current frame

    # The internal KV cache is automatically updated

Summary

• CameraHead and CameraCausalHead in lingbot_map/heads/camera_head.py predict camera extrinsics (translation, quaternion, focal length) using iterative refinement of a dedicated camera token, with the causal variant adding KV-cache support for streaming video.
• DPTHead in lingbot_map/heads/dpt_head.py generates dense per-pixel predictions (depth, normals, confidence) through multi-scale feature fusion and learned up-sampling.
• Both heads utilize activation utilities from lingbot_map/heads/head_act.py (activate_pose and activate_head) to support flexible output parameterizations.
• The modular design allows the transformer backbone to feed aggregated tokens to task-specific heads without architectural changes.

Frequently Asked Questions

What is the difference between CameraHead and CameraCausalHead?

CameraHead processes each frame independently using iterative refinement of a pose token through a transformer trunk. CameraCausalHead extends this with KV-cache support and optional 3-D RoPE positional encodings, enabling efficient streaming inference where past attention computations are cached and reused, making it suitable for video sequences and real-time SLAM.

How does DPTHead handle multi-scale features?

DPTHead receives token maps from multiple transformer layers (specified by intermediate_layer_idx), projects them to 2-D feature maps, and applies a cascade of FeatureFusionBlock modules to merge four spatial scales. This multi-scale approach captures both fine details and global context for dense prediction tasks.

What activation functions are available for pose prediction?

The activate_pose function in lingbot_map/heads/head_act.py supports multiple activation schemes including linear, relu, and inv_log (inverse logarithmic), allowing independent activation strategies for translation components, quaternion rotations, and focal length parameters.

Where are the positional embeddings for dense prediction computed?

Spatial grid utilities in lingbot_map/heads/utils.py (functions like create_uv_grid and position_grid_to_embed) generate positional embeddings that DPTHead optionally applies to the projected 2-D feature maps before up-sampling and fusion.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}