Core Architecture of LingBot-Map: Geometric Context Transformers for Streaming 3D Reconstruction

LingBot-Map is built around a Geometric Context Transformer (GCT) that fuses visual features, 3-D positional encoding, and a causal attention mechanism within a three-component streaming architecture to enable real-time, long-sequence 3D reconstruction.

LingBot-Map is an open-source streaming 3D reconstruction system designed to process video frames in real time while maintaining geometric consistency. As implemented in Robbyant/lingbot-map, the core architecture of LingBot-Map centers on a Geometric Context Transformer (GCT) that intelligently manages GPU memory through a paged KV-cache mechanism. This design enables simultaneous depth estimation and camera pose refinement on sequences exceeding 10,000 frames.

Overview of the Core Architecture of LingBot-Map

The architecture consists of three tightly coupled components orchestrated by the GCTStream class in lingbot_map/models/gct_stream.py:

Feature Aggregator (AggregatorStream)

The AggregatorStream extracts multi-scale visual tokens from each frame and stores them in a paged KV-cache. According to the source code, _build_aggregator() returns an AggregatorStream configured with parameters like kv_cache_sliding_window and kv_cache_scale_frames. This module optionally applies 3-D RoPE (Rotary Positional Encoding) to maintain temporal consistency across frames.

Camera Head (CameraCausalHead)

The CameraCausalHead refines per-frame camera poses using accumulated tokens in a strictly causal manner. Implemented via _build_camera_head() in the same file, this component supports iterative refinement through the camera_num_iterations parameter and can apply 3-D RoPE specific to the camera branch. It receives the same KV-cache settings as the aggregator and responds to keyframe logic to limit computation.

Streaming Engine (GCTStream)

The GCTStream class orchestrates the aggregator and camera head, managing the KV-cache life-cycle including cleaning, skip-append logic for non-keyframes, and statistics collection. It provides both streaming and windowed inference modes via the inference_streaming() and inference_windowed() methods. The streaming method implements a two-phase pipeline that processes scale frames in batch before switching to per-frame causal processing.

How the Streaming Pipeline Works

The reconstruction process follows a five-phase pipeline that balances accuracy with memory efficiency:

1. Input preprocessing – Images are converted to tensors with shape [S, 3, H, W] and optionally resized via LingbotMapMethod._prepare_images.

2. Scale frames – The first N frames (default 8) are processed together using bidirectional attention via a scale token to establish initial scene scale and depth estimates.

3. KV-cache streaming – Subsequent frames are processed causally. When keyframe_interval=1, every frame is cached; with keyframe_interval>1, only every k-th frame is stored. Non-keyframes trigger _set_skip_append(True) to discard KV after the forward pass, drastically reducing memory usage.

4. Windowed mode – For sequences exceeding the training range, the KV cache evicts old entries using a sliding window (window_size, overlap_size). The inference_windowed() method in benchmark/methods/lingbot_map.py wraps this logic and handles flow-based keyframe selection.

5. Post-processing – Raw pose encodings (pose_enc) are decoded into extrinsic/intrinsic matrices using pose_encoding_to_extri_intri, depth maps are reshaped, and optional confidence maps are attached.

Key Implementation Files

The following files define the core architecture of LingBot-Map:

• lingbot_map/models/gct_stream.py – Contains the GCTStream class, AggregatorStream, CameraCausalHead, and the KV-cache management logic.

• lingbot_map/models/gct_stream_window.py – Variant implementation supporting sliding-window KV eviction for ultra-long sequences.

• benchmark/methods/lingbot_map.py – Benchmark adapter that exposes the model to evaluation pipelines, handling checkpoint loading and result formatting.

• demo_render/batch_demo.py – End-to-end offline rendering pipeline demonstrating the streaming engine on video sequences.

• README.md – High-level documentation and quick-start commands.

Practical Usage Examples

Interactive Streaming Demo

Run the model on an image sequence with adjustable keyframe intervals to control memory usage:

python demo.py \
    --model_path /path/to/lingbot-map-long.pt \
    --image_folder example/oxford \
    --mask_sky \
    --keyframe_interval 2

This creates a LingbotMapMethod in streaming mode and calls inference_streaming(), caching only every second frame.

Programmatic Inference

Instantiate the GCTStream class directly for custom pipelines:

import torch
from lingbot_map.models.gct_stream import GCTStream

model = GCTStream(
    img_size=518,
    patch_size=14,
    enable_3d_rope=True,
    kv_cache_sliding_window=64,
    kv_cache_scale_frames=8,
    use_sdpa=False,  # FlashInfer backend

)

ckpt = torch.load("lingbot-map-long.pt", map_location="cpu")
model.load_state_dict(ckpt["model"], strict=False)
model.eval().cuda()

# Prepare a list of RGB frames (HxWx3 uint8)

rgb_list = [...]  # e.g. from a video decoder

imgs = torch.stack([torchvision.transforms.ToTensor()(im) for im in rgb_list]).cuda()

preds = model.inference_streaming(
    imgs,
    num_scale_frames=8,
    keyframe_interval=4,
    output_device=torch.device("cpu")
)

Windowed Inference for Long Sequences

For videos exceeding 10,000 frames, use the windowed variant:

from lingbot_map.models.gct_stream_window import GCTStream

model = GCTStream(
    img_size=518,
    patch_size=14,
    enable_3d_rope=True,
    kv_cache_sliding_window=64,
    kv_cache_scale_frames=8,
    use_sdpa=False,
)

preds = model.inference_windowed(
    imgs,
    window_size=128,
    overlap_size=8,
    keyframe_interval=13,
    flow_threshold=0.0,
    max_non_keyframe_gap=30,
)

This automatically resets the KV cache after each window, enabling stable inference beyond the 320-frame RoPE training limit.

Benchmark Adapter Integration

Use the benchmark wrapper for standardized evaluation:

from benchmark.methods.lingbot_map import LingbotMapMethod

method = LingbotMapMethod(
    checkpoint="lingbot-map-long.pt",
    device="cuda",
    mode="streaming",
    keyframe_interval="auto",
)

# `gt_artifact` is a benchmark dataset object providing `load_rgb_list()`

results = method.process_scene(gt_artifact)

The process_scene() method internally calls _prepare_images(), _run_inference(), and _process_outputs() to return evaluation-compatible results.

Offline Rendering Pipeline

Process long videos end-to-end with the rendering script:

python demo_render/batch_demo.py \
    --video_path /data/demo_videos/indoor_travel.MP4 \
    --output_folder /data/outputs/indoor_travel/ \
    --model_path /path/to/lingbot-map.pt \
    --config demo_render/config/indoor.yaml \
    --mode windowed --window_size 128 \
    --keyframe_interval 13 --overlap_keyframes 8 \
    --mask_sky \
    --save_predictions

This script writes an MP4 fly-through with optional sky-mask visualizations.

Summary

• LingBot-Map implements a Geometric Context Transformer (GCT) combining an AggregatorStream, CameraCausalHead, and GCTStream engine.
• KV-cache management with configurable keyframe_interval bounds memory usage by caching only essential frames.
• 3-D Rotary Positional Encoding (RoPE) maintains geometric consistency across the temporal dimension.
• Dual inference modes—streaming for real-time processing and windowed for sequences exceeding 10,000 frames—provide flexibility for diverse deployment scenarios.
• All core functionality resides in lingbot_map/models/gct_stream.py and is exposed via the LingbotMapMethod benchmark adapter.

Frequently Asked Questions

What is the Geometric Context Transformer in LingBot-Map?

The Geometric Context Transformer (GCT) is the central neural architecture that fuses visual features with 3-D positional information through causal attention mechanisms. It processes video streams frame-by-frame rather than requiring complete sequences upfront, enabling the real-time 3D reconstruction capabilities that define the core architecture of LingBot-Map.

How does LingBot-Map prevent GPU memory overflow on long videos?

The system uses a paged KV-cache with a configurable keyframe_interval parameter that restricts caching to every k-th frame. Additionally, the inference_windowed() method implements sliding-window eviction (window_size and overlap_size) to reset the cache periodically, allowing processing of sequences that far exceed the 320-frame training limit.

What distinguishes the streaming mode from the windowed inference mode?

Streaming mode processes frames sequentially with a monotonically growing KV-cache (limited by keyframe sampling), ideal for real-time applications. Windowed mode divides sequences into overlapping chunks, completely resetting the KV cache between windows to handle arbitrarily long videos while maintaining geometric consistency through shared keyframes.

Where is the camera pose refinement logic implemented?

Camera pose refinement resides in the CameraCausalHead class, created by _build_camera_head() in lingbot_map/models/gct_stream.py. This head receives aggregated tokens from the KV-cache and applies causal attention with optional 3-D RoPE to iteratively refine extrinsic and intrinsic camera parameters using the camera_num_iterations setting.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}