Core Architecture of LingBot-Map for Streaming 3D Reconstruction: Feed‑Forward Transformer with Paged KV‑Cache

LingBot-Map performs streaming 3D reconstruction using a feed‑forward geometric‑context transformer that maintains a paged KV‑cache of past token embeddings, enabling constant‑memory processing of arbitrarily long video streams through selective keyframe caching and automatic windowing.

The Robbyant/lingbot‑map repository implements a novel approach to real‑time 3D scene reconstruction that processes video incrementally rather than in batches. Unlike traditional structure‑from‑motion pipelines that require entire sequences to reside in memory, this system handles streaming input via a specialized transformer architecture. The core architecture of LingBot‑Map for streaming 3D reconstruction employs six tightly integrated components that maintain geometric coherence without recurrent neural network states.

Streaming Transformer Model (GCTStream)

At the heart of the system lies GCTStream, a feed‑forward geometric‑context transformer implemented in lingbot_map/models/gct_stream_window_v2.py. This model processes each incoming frame while re‑using cached key and value pairs from previous frames, eliminating the need for backpropagation through time or hidden state recurrence.

The transformer operates in a purely feed‑forward manner during inference. Each frame's features attend to past context through the KV‑cache rather than through recurrent connections, allowing the model to scale to videos of any length without increasing computational complexity per frame.

Token Aggregation Pipeline (AggregatorStream)

The AggregatorStream class, defined in lingbot_map/aggregator/stream.py, constructs the input token sequence for every frame. This component performs three critical functions:

• Patch embedding of RGB or RGB‑D frames into visual tokens.
• Insertion of anchor tokens that provide global scene context persisting across processing windows.
• Routing of pose‑reference tokens that supply camera pose history to the refinement heads.

By managing the token composition and cache interface, the aggregator ensures that the transformer receives properly formatted input while maintaining the separation between visual features and geometric metadata.

FlashInfer KV‑Cache Manager

Memory efficiency relies on the FlashInferKVCacheManager located in lingbot_map/layers/flashinfer_cache.py. This thin wrapper around FlashInfer’s paged attention kernels implements a sparse caching strategy:

• Keyframe selection: Only every N‑th frame (controlled by --keyframe_interval) stores its key and value tensors in the cache.
• Page eviction: Older slots are automatically evicted, bounding GPU memory usage regardless of video duration.
• O(1) attention cost: FlashInfer’s paged attention kernels enable constant‑time access to historical context per frame.

Non‑keyframes still generate reconstruction predictions but are not cached, significantly reducing memory pressure during long runs.

Camera Pose Refinement Head

Accurate camera tracking occurs through the camera head in lingbot_map/heads/camera_head.py. Rather than performing expensive bundle adjustment, this component refines each frame’s pose by attending to a sliding window of recent keyframes via the pose‑reference tokens inserted by the aggregator. This design enables accurate pose estimation while maintaining the streaming constraint of single‑pass processing.

Streaming Data Flow

The architectural components connect through a specific data pipeline that preserves temporal coherence:

1. Frame ingestion: Incoming RGB or RGB‑D frames enter the AggregatorStream.
2. Token construction: Visual patches combine with anchor and pose‑reference tokens to form the input sequence.
3. Cache lookup: The FlashInferKVCacheManager retrieves relevant historical keyframes via paged memory access.
4. Transformer inference: GCTStream processes the current tokens against the cached context.
5. Output generation: The system emits dense depth maps and refined camera poses, with the camera head consuming the pose‑reference window for geometric consistency.

Incoming Frame → AggregatorStream → Token Embedding
                                        ↓
FlashInfer KV-Cache ←─────────── GCTStream → Reconstruction Output
   (Keyframes only)                    ↓
                              Camera Head (Pose Refinement)

Memory Management and Windowed Inference

Keyframe Interval and Selective Caching

The --keyframe_interval parameter (exposed in demo.py and handled by flashinfer_cache.py) controls cache sparsity. Setting this to 2 caches every second frame, while higher values reduce memory usage at the cost of temporal resolution. Frames between keyframes are still processed for immediate reconstruction output but do not persist in the KV‑cache.

Windowed Mode for Long Sequences

When sequences exceed the training RoPE (Rotary Position Embedding) range of approximately 320 frames, LingBot‑Map automatically activates windowed inference. This mechanism, implemented in demo.py through flags such as --mode windowed, --window_size, and --overlap_keyframes, operates as follows:

• The stream splits into overlapping chunks of configurable size (e.g., 128 keyframes).
• Each new window resets the KV‑cache and re‑initializes the anchor token to prevent positional drift.
• Overlapping keyframes (--overlap_keyframes 8) ensure smooth pose continuity between windows.

This windowing strategy prevents RoPE overflow while maintaining geometric consistency across window boundaries.

Practical Implementation Examples

The following snippets demonstrate how to invoke the streaming pipeline using the command‑line interface or direct API access.

Basic streaming reconstruction with keyframe interval of 2:

python demo.py \
    --image_folder example/university \
    --model_path /path/to/lingbot-map-long.pt \
    --keyframe_interval 2 \
    --mask_sky

Windowed inference for videos exceeding 3000 frames:

python demo.py \
    --video_path /data/long_walk.mp4 \
    --fps 10 \
    --model_path /path/to/lingbot-map-long.pt \
    --mode windowed \
    --window_size 128 \
    --overlap_keyframes 8 \
    --keyframe_interval 13 \
    --mask_sky

Direct Python API usage for custom pipelines:

import torch
from lingbot_map.models.gct_stream_window_v2 import GCTStream
from lingbot_map.aggregator.stream import AggregatorStream

model = GCTStream.load_from_checkpoint("lingbot-map-long.pt")
model.eval().cuda()

# Assume `frames` is a list of (RGB, depth) tensors

for i, (rgb, depth) in enumerate(frames):
    # Build token stream for the current frame

    stream = AggregatorStream()
    stream.add_frame(rgb, depth)
    
    # Forward pass returns dense depth + camera pose

    out = model(stream.tokens, stream.kv_cache)
    # out contains reconstruction predictions for frame i

Summary

• GCTStream (gct_stream_window_v2.py) provides the core feed‑forward transformer that processes frames using cached historical context.
• AggregatorStream (aggregator/stream.py) constructs token sequences with anchor and pose‑reference tokens for geometric coherence.
• FlashInferKVCacheManager (layers/flashinfer_cache.py) implements paged attention for constant‑memory storage of keyframe embeddings.
• Keyframe intervals and windowed mode (demo.py) enable processing of arbitrarily long videos by splitting streams at the ~320‑frame RoPE boundary while maintaining overlap for continuity.
• Camera head (heads/camera_head.py) refines poses using a local window of recent keyframes, avoiding bundle adjustment bottlenecks.

Frequently Asked Questions

How does LingBot-Map maintain constant memory usage during streaming?

The architecture stores only keyframes (configurable via --keyframe_interval) in the paged KV‑cache managed by FlashInferKVCacheManager. When the cache reaches capacity, older pages are evicted, ensuring that GPU memory remains bounded regardless of video length. Non‑keyframe predictions are computed but not cached.

What is the purpose of the anchor token in the architecture?

The anchor token provides a global scene reference that persists across processing windows. Defined in lingbot_map/aggregator/stream.py, this special token allows the model to maintain consistent coordinate frames and scene semantics even when the KV‑cache resets during window transitions.

How does windowed inference prevent drift in long videos?

When sequences exceed the ~320‑frame RoPE limit, the system initiates a new window, resets the KV‑cache, and re‑initializes anchor tokens. The --overlap_keyframes parameter ensures that the last N keyframes of the previous window carry forward into the new window, providing temporal continuity that prevents geometric drift.

What is the difference between keyframes and non-keyframes in the KV-cache?

Keyframes (every N‑th frame where N equals --keyframe_interval) have their key and value embeddings stored in the FlashInfer paged cache for future attention. Non‑keyframes are processed through the transformer to generate immediate reconstruction outputs but are immediately discarded from memory, reducing storage requirements while maintaining inference speed.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}