Example Configurations for Using the Different Heads in LingBot-Map

LingBot-Map supports three prediction heads—camera, point, and depth—that you can enable, tune, or disable independently via command-line flags or YAML configuration files.

LingBot-Map is an open-source visual localization framework built around a shared transformer backbone that powers multiple task-specific prediction heads. Configuring these different heads allows you to trade off between computational cost and output richness, running anything from lightweight pose estimation to full dense reconstruction. Below are practical example configurations derived from the Robbyant/lingbot-map source code, showing exact file paths, class names, and runnable code snippets.

Where the Heads Are Defined

LingBot-Map implements three primary heads in the lingbot_map/heads/ directory. Each head receives aggregated tokens from the shared transformer trunk and runs its own specialized branch to produce final predictions.

According to the source code in lingbot_map/models/gct_base.py (lines 90-97), the model initializes heads conditionally:

self.camera_head = self._build_camera_head() if enable_camera else None
self.point_head   = self._build_point_head()   if enable_point   else None
self.depth_head   = self._build_depth_head()   if enable_depth   else None

The specific implementations are:

• CameraHead: Located in lingbot_map/heads/camera_head.py (lines 23-46), this class builds a transformer trunk and pose-branch MLP for iterative camera pose refinement. The CameraCausalHead variant (lines 57-89) adds KV-cache handling for streaming inference.
• DPTHead (Depth): Implemented in lingbot_map/heads/dpt_head.py (lines 21-38) using the dense prediction transformer architecture.
• PointHead: Defined in lingbot_map/heads/point_head.py, following the same construction pattern as the camera head.

How to Enable and Disable Heads

You control which heads are active using either command-line arguments for interactive demos or YAML presets for batch rendering.

Interactive Demo Flags

The demo.py script exposes boolean flags that toggle heads at runtime:

• --enable_camera – Activates the camera head for pose estimation.
• --enable_point – Activates the point-cloud head for 3-D point prediction.
• --enable_depth – Activates the depth head for dense depth mapping.
• --camera_num_iterations N – Sets refinement steps for the camera head (default 4).
• --keyframe_interval K – Controls KV-cache size for streaming camera inference.

Example commands from the repository documentation:

# Camera-only pose estimation

python demo.py --model_path /path/to/lingbot-map-long.pt \
    --image_folder example/courthouse --mask_sky --enable_camera

# Camera + depth for outdoor scenes

python demo.py --model_path /path/to/lingbot-map-long.pt \
    --image_folder example/oxford --mask_sky \
    --enable_camera --enable_depth

YAML Configuration for Batch Rendering

For offline video rendering via demo_render/batch_demo.py, heads are configured in YAML files located in demo_render/config/. Each preset contains a heads section:

# demo_render/config/indoor.yaml

heads:
  camera: true
  point:  true
  depth:  false

The CLI respects these YAML values unless explicitly overridden with --enable_* flags.

Practical Code Examples

These snippets demonstrate how to instantiate the model classes with specific head combinations. All examples assume installation via pip install -e ..

Camera-Only Inference (Streaming)

For long video sequences, use the GCTStream class with CameraCausalHead to maintain a KV cache:

import torch
from lingbot_map.models.gct_stream import GCTStream

model = GCTStream(
    enable_camera=True,
    enable_point=False,
    enable_depth=False,
    camera_head_kwargs=dict(
        num_iterations=2,        # faster inference

        keyframe_interval=2,     # reduce memory usage

    ),
).eval()

# tokens from backbone: list of [B, S, C] tensors

tokens = [torch.randn(1, 32, 2048)]
poses = model(tokens)[0]   # returns list of pose predictions per iteration

print(poses[-1].shape)       # (1, 9) → translation + quaternion + FOV

As implemented in gct_stream.py (lines 18-23), the enable_camera parameter triggers construction of the causal head variant, while trunk_fn in camera_head.py (lines 96-104) handles the iterative refinement.

Depth-Only Inference (Batch)

For batch processing without pose estimation:

import torch
from lingbot_map.models.gct_base import GCTBase

model = GCTBase(
    enable_camera=False,
    enable_point=False,
    enable_depth=True,
).eval()

images = torch.randn(1, 8, 3, 224, 224)   # B, S, C, H, W

tokens = [torch.randn(1, 8, 2048)]       # aggregated tokens

depth, conf = model(tokens, images, patch_start_idx=0)
print(depth.shape)   # (1, 8, 1, 224, 224)

The _build_depth_head method in gct_base.py (lines 102-105) instantiates the DPTHead, which executes in the forward pass around lines 190-200.

Full Pipeline Configuration

To run all three heads simultaneously with sliding-window support for long sequences:

import torch
from lingbot_map.models.gct_stream_window import GCTStreamWindow

model = GCTStreamWindow(
    enable_camera=True,
    enable_point=True,
    enable_depth=True,
    sliding_window_size=128,
    keyframe_interval=2,
).eval()

# Process 100-frame video

images = torch.randn(1, 100, 3, 224, 224)
tokens = [torch.randn(1, 100, 2048)]

poses, points, depth = model(tokens, images, patch_start_idx=0)
print(poses[-1].shape)   # (1, 9)

print(points[-1].shape)  # (1, 3, N)

print(depth.shape)       # (1, 100, 1, 224, 224)

The gct_stream_window.py implementation (lines 19-27) manages the KV cache and sliding window logic, while the forward method (lines 120-150) sequences calls to all three heads.

Summary

• LingBot-Map provides three interchangeable heads—camera, point, and depth—defined in lingbot_map/heads/camera_head.py, point_head.py, and dpt_head.py.
• Enable heads via CLI flags (--enable_camera, --enable_point, --enable_depth) in demo.py or through the heads: block in YAML configs for batch rendering.
• Tune camera head performance using num_iterations (accuracy vs. speed) and keyframe_interval (memory usage).
• Use streaming variants (GCTStream, GCTStreamWindow) for long videos to leverage KV-caching in the causal camera head.
• Combine heads arbitrarily to match your pipeline needs, from lightweight pose-only tracking to full 3-D reconstruction with dense depth and point clouds.

Frequently Asked Questions

How do I run only the camera head for pose estimation?

Pass --enable_camera without the other enable flags when running demo.py, or construct GCTStream with enable_camera=True and enable_point=False, enable_depth=False. This configuration minimizes memory usage and inference time while outputting only camera poses (translation, quaternion, and FOV) as a (1, 9) tensor.

Can I activate the depth head without the camera head?

Yes. Set enable_camera=False and enable_depth=True when instantiating GCTBase or GCTStream. The depth head (DPTHead) operates independently on the transformer backbone tokens and returns dense depth maps regardless of whether pose estimation is active.

What is the difference between CameraHead and CameraCausalHead?

CameraHead in camera_head.py (lines 23-46) is designed for batch processing where all frames are available simultaneously. CameraCausalHead (lines 57-89) extends this with KV-cache management, allowing the model to process streaming video efficiently by caching only keyframe tokens, as controlled by the keyframe_interval parameter.

How do I optimize memory usage for long video sequences?

Use the GCTStreamWindow class with a reduced sliding_window_size (e.g., 128 frames) and increase the keyframe_interval to 2 or 4. Enable only the heads you need—typically just enable_camera—and set num_iterations to 2 instead of the default 4 to reduce the number of refinement steps per frame.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}