# Prediction Heads in LingBot-Map: Camera Extrinsics and Dense Per-Pixel Outputs

> Explore the prediction heads in LingBot-Map. Learn how CameraHead estimates camera pose and DPTHead predicts dense depth and normals for precise spatial outputs.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: deep-dive
- Published: 2026-07-03

---

**LingBot-Map uses two specialized prediction heads—CameraHead for camera pose estimation and DPTHead for dense depth and normal prediction—to convert transformer token streams into concrete spatial outputs.**

LingBot-Map, available in the Robbyant/lingbot-map repository, implements a modular "head" architecture that bridges the gap between transformer backbones and task-specific outputs. These prediction heads transform aggregated token representations into either camera extrinsics for SLAM-style odometry or dense per-pixel maps for scene understanding.

## CameraHead and CameraCausalHead

The `CameraHead` family, defined in [`lingbot_map/heads/camera_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/camera_head.py), handles camera extrinsics prediction, outputting translation vectors, quaternion rotations, and focal length or field-of-view parameters.

### Architecture and Design

The `CameraHead` class processes a dedicated camera token through a small transformer "trunk" with configurable depth. The design employs an **empty pose token** as the initial input, iteratively refining the pose encoding across multiple iterations. Key implementation details include:

- **Adaptive layer normalization**: Token features are modulated with learnable shift, scale, and gate parameters.
- **Flexible activation**: Final outputs pass through `activate_pose` from [`lingbot_map/heads/head_act.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/head_act.py), supporting linear, ReLU, inverse-log, and other activation schemes.
- **Multi-iteration refinement**: The head can run multiple refinement steps (e.g., `num_iterations=4`) to progressively improve pose estimates.

### Streaming Inference with CameraCausalHead

For video sequences, `CameraCausalHead` extends the base architecture with KV-cache support, enabling **streaming inference** without recomputing past attention. As implemented in the same file, this variant maintains a sliding-window cache (`kv_cache`) and supports optional 3-D RoPE positional encodings. This allows the model to process frames sequentially, making it suitable for real-time SLAM applications.

## DPTHead for Dense Per-Pixel Prediction

The `DPTHead` class in [`lingbot_map/heads/dpt_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/dpt_head.py) generates dense predictions such as depth maps, surface normals, or confidence maps from multi-scale transformer features.

### Multi-Scale Feature Fusion

The head operates on token layers extracted at different depths of the backbone (configurable via `intermediate_layer_idx`). The pipeline:

1. Projects token maps to 2-D feature maps using learned projections.
2. Applies optional positional embeddings via utilities in [`lingbot_map/heads/utils.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/utils.py) (functions like `create_uv_grid` and `position_grid_to_embed`).
3. Up-samples features through learned transpose convolutions.
4. Fuses four spatial scales using `FeatureFusionBlock` modules in a scratch network configuration.

Final activations are handled by `activate_head` in [`head_act.py`](https://github.com/Robbyant/lingbot-map/blob/main/head_act.py), with support for `norm_exp`, `exp`, `relu`, and `inv_log` modes depending on whether the output represents depth or confidence.

## Activation Utilities

Both heads rely on shared activation functions defined in [`lingbot_map/heads/head_act.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/head_act.py):

- **`activate_pose`**: Applies component-specific activation to translation, quaternion, and focal length parameters.
- **`activate_head`**: Converts raw network outputs into 3-D points and confidence values with configurable activation schemes.

These utilities enable rapid experimentation with different output parameterizations without modifying the head implementations.

## Implementation Examples

### Camera Pose Prediction

```python
import torch
from lingbot_map.heads.camera_head import CameraHead

# Simulated aggregated token list (output of the backbone)

# Shape: [B, N, C] where N includes a dedicated camera token at index 0

tokens = torch.randn(2, 128, 2048)          # B=2, N=128, C=2048

aggregated_tokens = [tokens]                # list of token tensors from each stage

camera_head = CameraHead(
    dim_in=2048,
    trunk_depth=4,
    pose_encoding_type="absT_quaR_FoV",
    num_heads=16,
    mlp_ratio=4,
    trans_act="linear",
    quat_act="linear",
    fl_act="relu",
)

# Predict camera pose with four refinement iterations

pose_predictions = camera_head(aggregated_tokens, num_iterations=4)

# `pose_predictions` is a list of length 4 where each entry has shape

# (B, 1, 9) → [translation(3), quaternion(4), focal‑len/FoV(2)]

print(pose_predictions[0].shape)   # torch.Size([2, 1, 9])

```

### Dense Depth Prediction

```python
import torch
from lingbot_map.heads.dpt_head import DPTHead

# Assume we have 4 token tensors from different transformer layers,

# each with shape (B, S, C) where S includes patch tokens after the camera token.

tokens_layer0 = torch.randn(2, 64, 2048)
tokens_layer1 = torch.randn(2, 64, 2048)
tokens_layer2 = torch.randn(2, 64, 2048)
tokens_layer3 = torch.randn(2, 64, 2048)

aggregated_tokens = [tokens_layer0, tokens_layer1, tokens_layer2, tokens_layer3]

# Dummy image batch (required only for spatial dimensions)

images = torch.randn(2, 1, 3, 224, 224)   # B, S, C, H, W

# Index where patch tokens start (camera token occupies index 0)

patch_start_idx = 1

dpt_head = DPTHead(
    dim_in=2048,
    patch_size=14,
    output_dim=1,             # 1‑channel depth map

    activation="norm_exp",
    conf_activation="expp1",
    features=256,
    out_channels=[256, 512, 1024, 1024],
    intermediate_layer_idx=[0, 1, 2, 3],
)

depth, confidence = dpt_head(aggregated_tokens, images, patch_start_idx)
print(depth.shape)        # torch.Size([2, 1, 1, 224, 224])

print(confidence.shape)   # torch.Size([2, 1, 1, 224, 224])

```

### Streaming Camera Inference

```python
from lingbot_map.heads.camera_head import CameraCausalHead

causal_head = CameraCausalHead(
    dim_in=2048,
    trunk_depth=4,
    num_iterations=3,
    enable_3d_rope=False,
    sliding_window_size=64,
)

# Process frames one by one (or in small batches)

for frame_idx in range(num_frames):
    pose_list = causal_head(aggregated_tokens, causal_inference=True)
    # pose_list contains refined poses for the current frame

    # The internal KV cache is automatically updated

```

## Summary

- **CameraHead** and **CameraCausalHead** in [`lingbot_map/heads/camera_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/camera_head.py) predict camera extrinsics (translation, quaternion, focal length) using iterative refinement of a dedicated camera token, with the causal variant adding KV-cache support for streaming video.
- **DPTHead** in [`lingbot_map/heads/dpt_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/dpt_head.py) generates dense per-pixel predictions (depth, normals, confidence) through multi-scale feature fusion and learned up-sampling.
- Both heads utilize activation utilities from [`lingbot_map/heads/head_act.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/head_act.py) (`activate_pose` and `activate_head`) to support flexible output parameterizations.
- The modular design allows the transformer backbone to feed aggregated tokens to task-specific heads without architectural changes.

## Frequently Asked Questions

### What is the difference between CameraHead and CameraCausalHead?

**CameraHead** processes each frame independently using iterative refinement of a pose token through a transformer trunk. **CameraCausalHead** extends this with KV-cache support and optional 3-D RoPE positional encodings, enabling efficient streaming inference where past attention computations are cached and reused, making it suitable for video sequences and real-time SLAM.

### How does DPTHead handle multi-scale features?

`DPTHead` receives token maps from multiple transformer layers (specified by `intermediate_layer_idx`), projects them to 2-D feature maps, and applies a cascade of `FeatureFusionBlock` modules to merge four spatial scales. This multi-scale approach captures both fine details and global context for dense prediction tasks.

### What activation functions are available for pose prediction?

The `activate_pose` function in [`lingbot_map/heads/head_act.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/head_act.py) supports multiple activation schemes including `linear`, `relu`, and `inv_log` (inverse logarithmic), allowing independent activation strategies for translation components, quaternion rotations, and focal length parameters.

### Where are the positional embeddings for dense prediction computed?

Spatial grid utilities in [`lingbot_map/heads/utils.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/utils.py) (functions like `create_uv_grid` and `position_grid_to_embed`) generate positional embeddings that `DPTHead` optionally applies to the projected 2-D feature maps before up-sampling and fusion.