How LingBot-Map Handles Streaming 3D Reconstruction: Architecture and Implementation

LingBot-Map processes arbitrarily long video streams for 3D reconstruction using a feed-forward geometric-context transformer with a paged KV-cache that stores only keyframes, enabling constant GPU memory usage regardless of video length.

LingBot-Map is an open-source framework for real-time 3D scene reconstruction from video streams. Unlike batch-based methods that require entire videos to fit in memory, the repository implements a streaming architecture that processes frames incrementally while maintaining geometric consistency. This article examines the specific mechanisms—found in lingbot_map/models/gct_stream_window_v2.py and supporting modules—that enable efficient streaming 3D reconstruction without drift or memory overflow.

Core Streaming Architecture

The GCTStream Transformer Model

At the heart of the system is GCTStream, defined in lingbot_map/models/gct_stream_window_v2.py. This geometric-context transformer processes each incoming frame in a feed-forward manner while reusing cached key/value pairs from previous frames. The model maintains no recurrent hidden state; instead, it relies entirely on the KV-cache for temporal context, allowing each frame to be processed independently while still accessing historical geometric information.

Token Aggregation with AggregatorStream

The AggregatorStream class in lingbot_map/aggregator/stream.py constructs the input token sequence for each frame. It embeds visual patches and injects two critical token types: anchor tokens that provide global scene context, and pose-reference tokens that feed into the camera head. This module manages the interface between raw video frames and the transformer blocks, ensuring proper routing of cached embeddings to the attention mechanism.

Memory-Efficient Caching with FlashInfer

Paged KV-Cache Management

To prevent GPU memory from growing linearly with video length, LingBot-Map implements FlashInferKVCacheManager in lingbot_map/layers/flashinfer_cache.py. This wrapper around FlashInfer's paged attention kernels stores only keyframes—specifically every N-th frame as configured by the user. Non-keyframes generate predictions but are not cached, keeping memory usage roughly constant even for hours of footage.

Keyframe Interval Configuration

The --keyframe_interval parameter (exposed in demo.py) controls the sparsity of the cache. Setting this to 2 caches every second frame, while higher values reduce memory at the cost of temporal granularity. The cache manager automatically handles eviction of older slots when the allocation limit is reached, using FlashInfer's fast random access capabilities to maintain O(1) attention cost per frame.

Windowed Inference for Unlimited Stream Length

Handling RoPE Limitations

The transformer's Rotary Position Embedding (RoPE) is trained for a fixed context length of approximately 320 frames. When processing longer videos, LingBot-Map automatically splits the stream into overlapping windows. Each window resets the KV-cache and re-initializes anchor tokens, preventing positional drift that would otherwise corrupt the reconstruction.

Smooth Window Transitions

The --mode windowed flag in demo.py activates this behavior, using --window_size to specify the KV-cache slot limit and --overlap_keyframes to copy cached entries between adjacent windows. This overlap ensures pose continuity across window boundaries, eliminating the jitter typically associated with segment-based processing.

Camera Pose Refinement

The camera head in lingbot_map/heads/camera_head.py refines per-frame poses using a pose-reference window—a sliding buffer of recent keyframes. Rather than performing expensive bundle adjustment over the entire history, the head attends only to this local window, enabling accurate trajectory tracking while maintaining the streaming constraint of constant memory usage.

Practical Implementation Examples

Below are concrete commands and API calls for running streaming reconstruction.

Basic streaming with keyframe interval:

# CLI: Process image folder with every 2nd frame cached

python demo.py \
    --image_folder example/university \
    --model_path /path/to/lingbot-map-long.pt \
    --keyframe_interval 2 \
    --mask_sky

Windowed mode for long video sequences:

# CLI: Handle videos exceeding 3000 frames

python demo.py \
    --video_path /data/long_walk.mp4 \
    --fps 10 \
    --model_path /path/to/lingbot-map-long.pt \
    --mode windowed \
    --window_size 128 \
    --overlap_keyframes 8 \
    --keyframe_interval 13 \
    --mask_sky

Direct API integration:

import torch
from lingbot_map.models.gct_stream_window_v2 import GCTStream
from lingbot_map.aggregator.stream import AggregatorStream

model = GCTStream.load_from_checkpoint("lingbot-map-long.pt")
model.eval().cuda()

# Process frames in a custom pipeline

for i, (rgb, depth) in enumerate(frames):
    stream = AggregatorStream()
    stream.add_frame(rgb, depth)
    out = model(stream.tokens, stream.kv_cache)
    # out contains depth and camera pose for frame i

Summary

• LingBot-Map implements streaming 3D reconstruction through a feed-forward geometric-context transformer that uses a paged KV-cache rather than recurrent states.
• The FlashInferKVCacheManager in lingbot_map/layers/flashinfer_cache.py maintains constant memory by caching only keyframes at intervals specified by --keyframe_interval.
• AggregatorStream handles token preparation, injecting anchor and pose-reference tokens that provide global context and camera pose support.
• Windowed inference automatically segments long videos when approaching the RoPE limit (~320 frames), using overlapping keyframes to ensure continuity.
• The camera head refines poses using a sliding window of recent keyframes, avoiding global bundle adjustment while maintaining accuracy.

Frequently Asked Questions

How does LingBot-Map prevent memory overflow during long video processing?

LingBot-Map uses a paged KV-cache that stores only keyframes (every N-th frame) rather than every processed frame. The FlashInferKVCacheManager evicts older slots automatically, and the --keyframe_interval parameter lets users trade temporal resolution for memory efficiency. For sequences exceeding the training context length, windowed mode splits the video into overlapping segments.

What is the purpose of anchor tokens in the streaming pipeline?

Anchor tokens, managed by AggregatorStream in lingbot_map/aggregator/stream.py, provide global scene context that persists across processing windows. When the system resets the KV-cache for a new window (due to RoPE limitations), the anchor token re-initializes the global coordinate system, preventing drift in the reconstructed geometry.

Can LingBot-Map process live camera streams in real-time?

Yes. The architecture processes each frame independently using cached context from previous keyframes, with attention costs remaining O(1) per frame thanks to FlashInfer's paged attention kernels. The feed-forward design (no recurrent layers) and sparse keyframe caching enable near-real-time performance on modern GPUs.

How does windowed inference handle pose continuity between segments?

The --overlap_keyframes parameter copies cached KV entries from the end of one window to the beginning of the next. This overlap ensures that the camera head has access to recent pose history when processing the first frames of the new window, maintaining smooth trajectory estimation without gaps.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}