Lingbot-Map Architecture: Inside the Geometric Context Transformer

Lingbot-Map leverages a Geometric Context Transformer (GCT) that combines multi-scale visual feature aggregation, 3-D Rotary Positional Encoding (RoPE), and causal attention mechanisms to deliver real-time streaming 3-D reconstruction with bounded GPU memory.

The lingbot-map architecture powers a state-of-the-art streaming reconstruction system available in the Robbyant/lingbot-map repository. Unlike batch-processing approaches that require entire video sequences upfront, this architecture processes frames causally using a novel KV-cached transformer design, enabling real-time performance on long video streams while maintaining geometric consistency through 3-D positional encodings.

Core Architectural Components

The lingbot-map architecture consists of three tightly coupled modules that operate within a unified streaming framework defined in lingbot_map/models/gct_stream.py.

Feature Aggregator (AggregatorStream)

The _build_aggregator() method instantiates the AggregatorStream class, which extracts multi-scale visual tokens from each incoming frame. This component maintains tokens in a paged KV-cache and supports configurable parameters including kv_cache_sliding_window and kv_cache_scale_frames. It optionally applies 3-D RoPE (Rotary Positional Encoding) to enforce temporal consistency across the token sequence, ensuring geometric stability as the camera moves through the scene.

Camera Head (CameraCausalHead)

The _build_camera_head() method creates the CameraCausalHead, which refines per-frame camera poses using accumulated tokens from the aggregator. Operating strictly causally—attending only to past and current frames—this module supports iterative refinement via the camera_num_iterations parameter. It integrates with the keyframe logic to optimize pose estimation while managing computational overhead, and can utilize camera-specific 3-D RoPE configurations distinct from the visual branch.

Streaming Engine (GCTStream)

The GCTStream class orchestrates the aggregator and camera head, implementing the main inference loop in inference_streaming(). This engine manages the complete KV-cache lifecycle, including cleaning operations, "skip-append" logic for non-keyframes via _set_skip_append(True), and statistical tracking. It provides both streaming and windowed inference modes to handle variable sequence lengths efficiently without exhausting GPU memory.

How the Streaming Pipeline Works

The architecture implements a five-phase processing pipeline that balances reconstruction accuracy with memory efficiency.

1. Input Preprocessing: Images are converted to tensor format [S, 3, H, W] via LingbotMapMethod._prepare_images in benchmark/methods/lingbot_map.py, with optional resizing to maintain consistent input dimensions.

2. Scale Frame Initialization: The first N frames (default 8) are processed together using bidirectional attention through a special scale token. This establishes the initial scene scale and provides baseline depth and pose estimates that anchor the subsequent causal processing.

3. KV-Cache Streaming: After initialization, subsequent frames are processed causally. When keyframe_interval=1, every frame's key-value pairs are cached. With keyframe_interval > 1, only every k-th frame is stored; non-keyframes trigger _set_skip_append(True) to discard their KV tensors immediately after the forward pass, drastically reducing memory consumption during long sequences.

4. Windowed Mode for Long Sequences: For sequences exceeding the 320-frame RoPE training range, the inference_windowed() method (available in both gct_stream.py and gct_stream_window.py) implements sliding-window cache eviction using window_size and overlap_size parameters. The benchmark adapter wraps this functionality with flow-based keyframe selection to maintain continuity across window boundaries.

5. Post-Processing: Raw pose encodings (pose_enc) are decoded into extrinsic and intrinsic camera matrices via pose_encoding_to_extri_intri() in lingbot_map/utils/pose_enc.py, while depth maps are reshaped to match input resolutions and optional confidence maps are attached to the output dictionary.

Key Implementation Files

Understanding the lingbot-map architecture requires familiarity with these critical source files:

• lingbot_map/models/gct_stream.py: Contains the core GCTStream class, AggregatorStream, CameraCausalHead, and the KV-cache management logic including the keyframe interval mechanisms.
• lingbot_map/models/gct_stream_window.py: Specialized variant implementing sliding-window KV eviction for ultra-long sequences that exceed standard positional encoding limits.
• benchmark/methods/lingbot_map.py: Benchmark adapter that handles checkpoint loading, image preparation via _prepare_images(), inference routing through _run_inference(), and result formatting via _process_outputs().
• demo_render/batch_demo.py: End-to-end offline rendering pipeline demonstrating the streaming engine on long video sequences with configurable visualization options.

Practical Usage Examples

Running Streaming Inference

Execute real-time reconstruction with configurable keyframe intervals to control memory usage:

python demo.py \
    --model_path /path/to/lingbot-map-long.pt \
    --image_folder example/oxford \
    --mask_sky \
    --keyframe_interval 2

This command creates a LingbotMapMethod instance in streaming mode, where the KV-cache grows only on keyframes, maintaining bounded GPU memory usage suitable for real-time applications.

Programmatic Model Initialization

Instantiate the architecture directly for custom pipelines:

import torch
from lingbot_map.models.gct_stream import GCTStream

# Initialize streaming model with 3-D RoPE enabled

model = GCTStream(
    img_size=518,
    patch_size=14,
    enable_3d_rope=True,
    kv_cache_sliding_window=64,
    kv_cache_scale_frames=8,
    use_sdpa=False,  # Uses FlashInfer backend for fast KV paging

)

# Load checkpoint

ckpt = torch.load("lingbot-map-long.pt", map_location="cpu")
model.load_state_dict(ckpt["model"], strict=False)
model.eval().cuda()

# Process RGB frames

rgb_list = [...]  # List of HxWx3 uint8 images

imgs = torch.stack([torchvision.transforms.ToTensor()(im) for im in rgb_list]).cuda()

# Run inference with keyframe every 4 frames

preds = model.inference_streaming(
    imgs,
    num_scale_frames=8,
    keyframe_interval=4,
    output_device=torch.device("cpu")
)

# Decode camera parameters

from lingbot_map.utils.pose_enc import pose_encoding_to_extri_intri
extr, intr = pose_encoding_to_extri_intri(preds["pose_enc"], imgs.shape[-2:])

Windowed Inference for Extended Sequences

Handle videos exceeding 10,000 frames using sliding windows to prevent cache overflow:

from lingbot_map.models.gct_stream_window import GCTStream

model = GCTStream(
    img_size=518,
    patch_size=14,
    enable_3d_rope=True,
    kv_cache_sliding_window=64,
    kv_cache_scale_frames=8,
    use_sdpa=False,
)

# Windowed processing with automatic KV reset between windows

preds = model.inference_windowed(
    imgs,
    window_size=128,      # KV slots per window

    overlap_size=8,       # Shared keyframes between windows

    keyframe_interval=13,
    flow_threshold=0.0,
    max_non_keyframe_gap=30,
)

Benchmark Integration

Integrate with evaluation frameworks using the provided adapter:

from benchmark.methods.lingbot_map import LingbotMapMethod

method = LingbotMapMethod(
    checkpoint="lingbot-map-long.pt",
    device="cuda",
    mode="streaming",
    keyframe_interval="auto",  # Auto-select based on sequence length

)

results = method.process_scene(gt_artifact)

Summary

• The lingbot-map architecture centers on a Geometric Context Transformer (GCT) that enables real-time streaming 3-D reconstruction through causal attention mechanisms.
• Three core components—the AggregatorStream, CameraCausalHead, and GCTStream engine—work together in lingbot_map/models/gct_stream.py to process visual tokens and camera poses sequentially.
• KV-cache management with configurable keyframe_interval parameters and the _set_skip_append() method ensures GPU memory remains bounded during long sequences.
• Dual inference modes (inference_streaming() and inference_windowed()) support both low-latency real-time applications and offline processing of ultra-long videos exceeding the 320-frame RoPE training range.
• 3-D Rotary Positional Encoding maintains geometric consistency across the temporal dimension, distinguishing this architecture from standard 2-D vision transformers.

Frequently Asked Questions

What is the Geometric Context Transformer in lingbot-map?

The Geometric Context Transformer (GCT) is the core neural architecture that fuses multi-scale visual features with 3-D geometric information. It employs causal attention mechanisms and 3-D Rotary Positional Encoding (RoPE) to process video streams frame-by-frame while maintaining temporal consistency, as implemented in the GCTStream class within lingbot_map/models/gct_stream.py. This design allows the model to understand spatial relationships in dynamic scenes without requiring future frame information.

How does lingbot-map handle memory constraints during long video processing?

The architecture implements a paged KV-cache system with configurable eviction policies. Through the keyframe_interval parameter in inference_streaming(), non-keyframes can skip cache appending via _set_skip_append(True), reducing cache growth. For extremely long sequences, the inference_windowed() method in gct_stream_window.py provides sliding-window cache eviction with window_size and overlap_size controls, enabling processing of sequences that far exceed the model's 320-frame positional encoding limit.

What distinguishes the streaming mode from windowed inference?

Streaming mode (inference_streaming()) processes frames continuously with a monotonically growing KV-cache (constrained only by keyframe skipping), making it ideal for real-time applications requiring immediate output. Windowed inference (inference_windowed()) periodically resets the KV-cache using overlapping windows, enabling stable processing of hour-long videos that would otherwise exceed the model's positional encoding range or available GPU memory.

Where is the camera pose estimation logic located?

Camera pose refinement occurs in the CameraCausalHead class, constructed via _build_camera_head() in lingbot_map/models/gct_stream.py. This module processes tokens from the AggregatorStream using causal self-attention and iterative refinement controlled by camera_num_iterations. Final pose decoding from latent encodings to extrinsic/intrinsic matrices is handled by pose_encoding_to_extri_intri() in lingbot_map/utils/pose_enc.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}