Camera Head Models in camera_head.py: CameraHead, CameraCausalHead, and CameraDecoder Explained

The camera_head.py module implements three distinct camera heads: CameraHead for non-causal batch inference, CameraCausalHead for streaming video with KV-cache support, and CameraDecoder for lightweight feature projection.

The lingbot_map/heads/camera_head.py file defines the camera prediction components for the LingBot-Map architecture. These transformer-based heads estimate camera parameters—including translation, quaternion rotation, and field-of-view—through iterative refinement of learnable pose tokens, serving both batch and streaming inference scenarios.

Non-Causal Camera Head for Batch Processing

CameraHead Architecture

CameraHead serves as the default camera head for batch processing in the base GCT model. Located in lingbot_map/heads/camera_head.py, this class predicts camera parameters by iteratively refining a pose token through a stack of regular transformer blocks.

The head processes aggregated tokens from the backbone and performs multiple refinement iterations to predict translation, quaternion, and field-of-view values. According to the Robbyant/lingbot-map source code, this component is instantiated in lingbot_map/models/gct_base.py when enable_camera=True.

from lingbot_map.heads.camera_head import CameraHead

# Create a head that expects 2048-dim token vectors

cam_head = CameraHead(dim_in=2048, trunk_depth=4)

# Forward pass – aggregated_tokens is a list of token tensors from the backbone

pred_poses = cam_head(aggregated_tokens, num_iterations=4)  # → list of 4 pose encodings

Causal Streaming Camera Head

CameraCausalHead with KV-Cache Support

CameraCausalHead extends the base functionality for streaming video applications. This causal head adds KV-cache handling, sliding-window support, and optional 3-D RoPE positional encoding (WanRotaryPosEmbed) to enable frame-by-frame inference while maintaining past context.

The implementation supports streaming GCT models across three variants:

• lingbot_map/models/gct_stream.py
• lingbot_map/models/gct_stream_window.py
• lingbot_map/models/gct_stream_window_v2.py

Unlike the non-causal version, CameraCausalHead manages temporal dependencies through cached key-value states and sliding-window attention mechanisms, making it suitable for real-time video processing.

from lingbot_map.models.gct_stream import GCTStream

# The model automatically builds a CameraCausalHead when enable_camera=True

model = GCTStream(enable_camera=True, dim_in=2048, trunk_depth=4)

# Streaming inference: feed one frame at a time

for frame_tokens in video_frames:
    # aggregated_tokens is a list where the last entry contains the latest tokens

    poses = model(frame_tokens)          # → list of pose predictions for this frame

Lightweight Feature Decoder

CameraDecoder for Downstream Tasks

CameraDecoder provides a lightweight alternative for projecting hidden features into desired output shapes. This decoder builds a small stack of transformer blocks followed by a final linear projection, making it suitable for depth estimation or semantic mapping tasks rather than direct camera parameter regression.

Unlike the pose-specific heads, CameraDecoder is not tied to specific GCT model variants but remains available for custom pipelines requiring flexible feature decoding.

from lingbot_map.heads.camera_head import CameraDecoder
import torch

decoder = CameraDecoder(in_dim=1024, out_dim=9, dec_embed_dim=512, depth=3)

hidden = torch.randn(2, 5, 16, 1024)   # (B, V, P, C)

decoded = decoder(hidden)              # → (B, V, P, 9) pose encodings

Shared Design Patterns Across Camera Heads

All three camera heads in camera_head.py share a common architectural foundation:

Pose Token Initialization

Each head utilizes a learnable empty token (self.empty_pose_tokens) that expands to the batch size and embeds via self.embed_pose. This token serves as the initial state for iterative refinement.

Iterative Refinement Mechanism

The refinement process predicts delta poses (pred_pose_enc_delta) across multiple iterations, accumulating results before passing through activate_pose to enforce proper ranges for translation, quaternion, and field-of-view parameters.

Adaptive Layer Normalization

Modulation layers implement adaptive layer normalization (self.adaln_norm) with learned shift, scale, and gate parameters (self.poseLN_modulation), enabling dynamic feature adaptation during pose estimation.

Integration with GCT Model Variants

The camera heads integrate differently across the LingBot-Map model family:

• Base GCT (lingbot_map/models/gct_base.py): Uses CameraHead for batch processing of complete sequences
• Streaming GCT (lingbot_map/models/gct_stream.py): Employs CameraCausalHead for online inference
• Windowed Streaming (lingbot_map/models/gct_stream_window.py and gct_stream_window_v2.py): Leverages the causal head with sliding-window constraints for memory-efficient long-video processing

Summary

• CameraHead handles batch camera parameter prediction through iterative transformer refinement in non-causal settings
• CameraCausalHead enables real-time streaming with KV-cache, 3-D RoPE, and sliding-window attention for temporal consistency
• CameraDecoder provides lightweight feature projection for custom downstream tasks without pose-specific constraints
• All heads share pose token initialization, iterative refinement, and adaptive layer normalization patterns
• Implementation files include lingbot_map/heads/camera_head.py and model variants in lingbot_map/models/gct_*.py

Frequently Asked Questions

What is the difference between CameraHead and CameraCausalHead?

CameraHead processes entire sequences simultaneously without temporal constraints, making it suitable for batch inference where all frames are available upfront. CameraCausalHead adds causal masking, KV-cache management, and 3-D rotary positional encoding to support streaming inference where frames arrive sequentially, maintaining context from previous frames while preventing information leakage from future frames.

How does iterative refinement work in camera_head.py?

Each camera head initializes a learnable pose token that undergoes multiple transformation iterations. During each iteration, the model predicts a delta pose (pred_pose_enc_delta) that accumulates with previous predictions. After the final iteration, the activate_pose function constrains the values to valid ranges for translation vectors, quaternion rotations, and field-of-view parameters.

Where is CameraDecoder used in the LingBot-Map pipeline?

CameraDecoder operates as a general-purpose projection module for downstream tasks that require converting hidden transformer features into specific output dimensions. While CameraHead and CameraCausalHead integrate directly into the GCT model files (gct_base.py, gct_stream.py), the decoder remains available for custom implementations such as depth prediction or semantic segmentation heads that need lightweight feature transformation.

What camera parameters do these heads predict?

The camera heads predict translation (3D position), quaternion (orientation), and field-of-view (camera intrinsics). The activate_pose method ensures translation values, quaternion components, and FoV parameters remain within physically valid ranges through appropriate activation functions applied to the final pose encodings.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}