# How LingBot-Map Handles Streaming 3D Reconstruction: Architecture and Implementation

> Discover how LingBot-Map achieves efficient streaming 3D reconstruction. Learn about its geometric-context transformer and paged KV-cache for constant GPU memory use, even with long videos.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: architecture
- Published: 2026-07-05

---

**LingBot-Map processes arbitrarily long video streams for 3D reconstruction using a feed-forward geometric-context transformer with a paged KV-cache that stores only keyframes, enabling constant GPU memory usage regardless of video length.**

LingBot-Map is an open-source framework for real-time 3D scene reconstruction from video streams. Unlike batch-based methods that require entire videos to fit in memory, the repository implements a streaming architecture that processes frames incrementally while maintaining geometric consistency. This article examines the specific mechanisms—found in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py) and supporting modules—that enable efficient streaming 3D reconstruction without drift or memory overflow.

## Core Streaming Architecture

### The GCTStream Transformer Model

At the heart of the system is `GCTStream`, defined in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py). This **geometric-context transformer** processes each incoming frame in a feed-forward manner while reusing cached key/value pairs from previous frames. The model maintains no recurrent hidden state; instead, it relies entirely on the KV-cache for temporal context, allowing each frame to be processed independently while still accessing historical geometric information.

### Token Aggregation with AggregatorStream

The `AggregatorStream` class in [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py) constructs the input token sequence for each frame. It embeds visual patches and injects two critical token types: **anchor tokens** that provide global scene context, and **pose-reference tokens** that feed into the camera head. This module manages the interface between raw video frames and the transformer blocks, ensuring proper routing of cached embeddings to the attention mechanism.

## Memory-Efficient Caching with FlashInfer

### Paged KV-Cache Management

To prevent GPU memory from growing linearly with video length, LingBot-Map implements `FlashInferKVCacheManager` in [`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py). This wrapper around FlashInfer's paged attention kernels stores only **keyframes**—specifically every N-th frame as configured by the user. Non-keyframes generate predictions but are not cached, keeping memory usage roughly constant even for hours of footage.

### Keyframe Interval Configuration

The `--keyframe_interval` parameter (exposed in [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py)) controls the sparsity of the cache. Setting this to 2 caches every second frame, while higher values reduce memory at the cost of temporal granularity. The cache manager automatically handles eviction of older slots when the allocation limit is reached, using FlashInfer's fast random access capabilities to maintain O(1) attention cost per frame.

## Windowed Inference for Unlimited Stream Length

### Handling RoPE Limitations

The transformer's Rotary Position Embedding (RoPE) is trained for a fixed context length of approximately 320 frames. When processing longer videos, LingBot-Map automatically splits the stream into overlapping windows. Each window resets the KV-cache and re-initializes anchor tokens, preventing positional drift that would otherwise corrupt the reconstruction.

### Smooth Window Transitions

The `--mode windowed` flag in [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py) activates this behavior, using `--window_size` to specify the KV-cache slot limit and `--overlap_keyframes` to copy cached entries between adjacent windows. This overlap ensures pose continuity across window boundaries, eliminating the jitter typically associated with segment-based processing.

## Camera Pose Refinement

The camera head in [`lingbot_map/heads/camera_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/camera_head.py) refines per-frame poses using a **pose-reference window**—a sliding buffer of recent keyframes. Rather than performing expensive bundle adjustment over the entire history, the head attends only to this local window, enabling accurate trajectory tracking while maintaining the streaming constraint of constant memory usage.

## Practical Implementation Examples

Below are concrete commands and API calls for running streaming reconstruction.

Basic streaming with keyframe interval:

```python

# CLI: Process image folder with every 2nd frame cached

python demo.py \
    --image_folder example/university \
    --model_path /path/to/lingbot-map-long.pt \
    --keyframe_interval 2 \
    --mask_sky

```

Windowed mode for long video sequences:

```python

# CLI: Handle videos exceeding 3000 frames

python demo.py \
    --video_path /data/long_walk.mp4 \
    --fps 10 \
    --model_path /path/to/lingbot-map-long.pt \
    --mode windowed \
    --window_size 128 \
    --overlap_keyframes 8 \
    --keyframe_interval 13 \
    --mask_sky

```

Direct API integration:

```python
import torch
from lingbot_map.models.gct_stream_window_v2 import GCTStream
from lingbot_map.aggregator.stream import AggregatorStream

model = GCTStream.load_from_checkpoint("lingbot-map-long.pt")
model.eval().cuda()

# Process frames in a custom pipeline

for i, (rgb, depth) in enumerate(frames):
    stream = AggregatorStream()
    stream.add_frame(rgb, depth)
    out = model(stream.tokens, stream.kv_cache)
    # out contains depth and camera pose for frame i

```

## Summary

- **LingBot-Map** implements streaming 3D reconstruction through a feed-forward geometric-context transformer that uses a paged KV-cache rather than recurrent states.
- The `FlashInferKVCacheManager` in [`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py) maintains constant memory by caching only keyframes at intervals specified by `--keyframe_interval`.
- `AggregatorStream` handles token preparation, injecting anchor and pose-reference tokens that provide global context and camera pose support.
- Windowed inference automatically segments long videos when approaching the RoPE limit (~320 frames), using overlapping keyframes to ensure continuity.
- The camera head refines poses using a sliding window of recent keyframes, avoiding global bundle adjustment while maintaining accuracy.

## Frequently Asked Questions

### How does LingBot-Map prevent memory overflow during long video processing?

LingBot-Map uses a paged KV-cache that stores only keyframes (every N-th frame) rather than every processed frame. The `FlashInferKVCacheManager` evicts older slots automatically, and the `--keyframe_interval` parameter lets users trade temporal resolution for memory efficiency. For sequences exceeding the training context length, windowed mode splits the video into overlapping segments.

### What is the purpose of anchor tokens in the streaming pipeline?

Anchor tokens, managed by `AggregatorStream` in [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py), provide global scene context that persists across processing windows. When the system resets the KV-cache for a new window (due to RoPE limitations), the anchor token re-initializes the global coordinate system, preventing drift in the reconstructed geometry.

### Can LingBot-Map process live camera streams in real-time?

Yes. The architecture processes each frame independently using cached context from previous keyframes, with attention costs remaining O(1) per frame thanks to FlashInfer's paged attention kernels. The feed-forward design (no recurrent layers) and sparse keyframe caching enable near-real-time performance on modern GPUs.

### How does windowed inference handle pose continuity between segments?

The `--overlap_keyframes` parameter copies cached KV entries from the end of one window to the beginning of the next. This overlap ensures that the camera head has access to recent pose history when processing the first frames of the new window, maintaining smooth trajectory estimation without gaps.