# Camera Head Models in camera_head.py: CameraHead, CameraCausalHead, and CameraDecoder Explained

> Explore three camera head models in lingbot-map: CameraHead for batch inference, CameraCausalHead for streaming video, and CameraDecoder for lightweight feature projection.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: deep-dive
- Published: 2026-07-07

---

**The [`camera_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/camera_head.py) module implements three distinct camera heads: `CameraHead` for non-causal batch inference, `CameraCausalHead` for streaming video with KV-cache support, and `CameraDecoder` for lightweight feature projection.**

The [`lingbot_map/heads/camera_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/camera_head.py) file defines the camera prediction components for the LingBot-Map architecture. These transformer-based heads estimate camera parameters—including translation, quaternion rotation, and field-of-view—through iterative refinement of learnable pose tokens, serving both batch and streaming inference scenarios.

## Non-Causal Camera Head for Batch Processing

### CameraHead Architecture

**`CameraHead`** serves as the default camera head for batch processing in the base GCT model. Located in [`lingbot_map/heads/camera_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/camera_head.py), this class predicts camera parameters by iteratively refining a pose token through a stack of regular transformer blocks.

The head processes **aggregated tokens** from the backbone and performs multiple refinement iterations to predict translation, quaternion, and field-of-view values. According to the Robbyant/lingbot-map source code, this component is instantiated in [`lingbot_map/models/gct_base.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_base.py) when `enable_camera=True`.

```python
from lingbot_map.heads.camera_head import CameraHead

# Create a head that expects 2048-dim token vectors

cam_head = CameraHead(dim_in=2048, trunk_depth=4)

# Forward pass – aggregated_tokens is a list of token tensors from the backbone

pred_poses = cam_head(aggregated_tokens, num_iterations=4)  # → list of 4 pose encodings

```

## Causal Streaming Camera Head

### CameraCausalHead with KV-Cache Support

**`CameraCausalHead`** extends the base functionality for streaming video applications. This causal head adds **KV-cache handling**, **sliding-window support**, and optional **3-D RoPE positional encoding** (`WanRotaryPosEmbed`) to enable frame-by-frame inference while maintaining past context.

The implementation supports streaming GCT models across three variants:
- [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py)
- [`lingbot_map/models/gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py)  
- [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py)

Unlike the non-causal version, `CameraCausalHead` manages temporal dependencies through cached key-value states and sliding-window attention mechanisms, making it suitable for real-time video processing.

```python
from lingbot_map.models.gct_stream import GCTStream

# The model automatically builds a CameraCausalHead when enable_camera=True

model = GCTStream(enable_camera=True, dim_in=2048, trunk_depth=4)

# Streaming inference: feed one frame at a time

for frame_tokens in video_frames:
    # aggregated_tokens is a list where the last entry contains the latest tokens

    poses = model(frame_tokens)          # → list of pose predictions for this frame

```

## Lightweight Feature Decoder

### CameraDecoder for Downstream Tasks

**`CameraDecoder`** provides a lightweight alternative for projecting hidden features into desired output shapes. This decoder builds a small stack of transformer blocks followed by a final linear projection, making it suitable for depth estimation or semantic mapping tasks rather than direct camera parameter regression.

Unlike the pose-specific heads, `CameraDecoder` is not tied to specific GCT model variants but remains available for custom pipelines requiring flexible feature decoding.

```python
from lingbot_map.heads.camera_head import CameraDecoder
import torch

decoder = CameraDecoder(in_dim=1024, out_dim=9, dec_embed_dim=512, depth=3)

hidden = torch.randn(2, 5, 16, 1024)   # (B, V, P, C)

decoded = decoder(hidden)              # → (B, V, P, 9) pose encodings

```

## Shared Design Patterns Across Camera Heads

All three camera heads in [`camera_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/camera_head.py) share a common architectural foundation:

### Pose Token Initialization

Each head utilizes a **learnable empty token** (`self.empty_pose_tokens`) that expands to the batch size and embeds via `self.embed_pose`. This token serves as the initial state for iterative refinement.

### Iterative Refinement Mechanism

The refinement process predicts delta poses (`pred_pose_enc_delta`) across multiple iterations, accumulating results before passing through `activate_pose` to enforce proper ranges for translation, quaternion, and field-of-view parameters.

### Adaptive Layer Normalization

**Modulation layers** implement adaptive layer normalization (`self.adaln_norm`) with learned shift, scale, and gate parameters (`self.poseLN_modulation`), enabling dynamic feature adaptation during pose estimation.

## Integration with GCT Model Variants

The camera heads integrate differently across the LingBot-Map model family:

- **Base GCT** ([`lingbot_map/models/gct_base.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_base.py)): Uses `CameraHead` for batch processing of complete sequences
- **Streaming GCT** ([`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py)): Employs `CameraCausalHead` for online inference
- **Windowed Streaming** ([`lingbot_map/models/gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py) and [`gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream_window_v2.py)): Leverages the causal head with sliding-window constraints for memory-efficient long-video processing

## Summary

- **`CameraHead`** handles batch camera parameter prediction through iterative transformer refinement in non-causal settings
- **`CameraCausalHead`** enables real-time streaming with KV-cache, 3-D RoPE, and sliding-window attention for temporal consistency
- **`CameraDecoder`** provides lightweight feature projection for custom downstream tasks without pose-specific constraints
- All heads share **pose token initialization**, **iterative refinement**, and **adaptive layer normalization** patterns
- Implementation files include [`lingbot_map/heads/camera_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/camera_head.py) and model variants in `lingbot_map/models/gct_*.py`

## Frequently Asked Questions

### What is the difference between CameraHead and CameraCausalHead?

**`CameraHead`** processes entire sequences simultaneously without temporal constraints, making it suitable for batch inference where all frames are available upfront. **`CameraCausalHead`** adds causal masking, KV-cache management, and 3-D rotary positional encoding to support streaming inference where frames arrive sequentially, maintaining context from previous frames while preventing information leakage from future frames.

### How does iterative refinement work in camera_head.py?

Each camera head initializes a learnable **pose token** that undergoes multiple transformation iterations. During each iteration, the model predicts a delta pose (`pred_pose_enc_delta`) that accumulates with previous predictions. After the final iteration, the `activate_pose` function constrains the values to valid ranges for translation vectors, quaternion rotations, and field-of-view parameters.

### Where is CameraDecoder used in the LingBot-Map pipeline?

**`CameraDecoder`** operates as a general-purpose projection module for downstream tasks that require converting hidden transformer features into specific output dimensions. While `CameraHead` and `CameraCausalHead` integrate directly into the GCT model files ([`gct_base.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_base.py), [`gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream.py)), the decoder remains available for custom implementations such as depth prediction or semantic segmentation heads that need lightweight feature transformation.

### What camera parameters do these heads predict?

The camera heads predict **translation** (3D position), **quaternion** (orientation), and **field-of-view** (camera intrinsics). The `activate_pose` method ensures translation values, quaternion components, and FoV parameters remain within physically valid ranges through appropriate activation functions applied to the final pose encodings.