# Core Architecture of LingBot-Map: Geometric Context Transformers for Streaming 3D Reconstruction

> Explore LingBot-Map's core architecture, a Geometric Context Transformer for real-time, streaming 3D reconstruction. Learn how it fuses visual, positional, and causal attention.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: architecture
- Published: 2026-07-05

---

**LingBot-Map is built around a Geometric Context Transformer (GCT) that fuses visual features, 3-D positional encoding, and a causal attention mechanism within a three-component streaming architecture to enable real-time, long-sequence 3D reconstruction.**

LingBot-Map is an open-source streaming 3D reconstruction system designed to process video frames in real time while maintaining geometric consistency. As implemented in Robbyant/lingbot-map, the core architecture of LingBot-Map centers on a Geometric Context Transformer (GCT) that intelligently manages GPU memory through a paged KV-cache mechanism. This design enables simultaneous depth estimation and camera pose refinement on sequences exceeding 10,000 frames.

## Overview of the Core Architecture of LingBot-Map

The architecture consists of three tightly coupled components orchestrated by the `GCTStream` class in [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py):

### Feature Aggregator (AggregatorStream)

The **AggregatorStream** extracts multi-scale visual tokens from each frame and stores them in a **paged KV-cache**. According to the source code, `_build_aggregator()` returns an `AggregatorStream` configured with parameters like `kv_cache_sliding_window` and `kv_cache_scale_frames`. This module optionally applies **3-D RoPE** (Rotary Positional Encoding) to maintain temporal consistency across frames.

### Camera Head (CameraCausalHead)

The **CameraCausalHead** refines per-frame camera poses using accumulated tokens in a strictly causal manner. Implemented via `_build_camera_head()` in the same file, this component supports iterative refinement through the `camera_num_iterations` parameter and can apply 3-D RoPE specific to the camera branch. It receives the same KV-cache settings as the aggregator and responds to keyframe logic to limit computation.

### Streaming Engine (GCTStream)

The **GCTStream** class orchestrates the aggregator and camera head, managing the KV-cache life-cycle including cleaning, skip-append logic for non-keyframes, and statistics collection. It provides both **streaming** and **windowed** inference modes via the `inference_streaming()` and `inference_windowed()` methods. The streaming method implements a two-phase pipeline that processes scale frames in batch before switching to per-frame causal processing.

## How the Streaming Pipeline Works

The reconstruction process follows a five-phase pipeline that balances accuracy with memory efficiency:

1. **Input preprocessing** – Images are converted to tensors with shape `[S, 3, H, W]` and optionally resized via `LingbotMapMethod._prepare_images`.

2. **Scale frames** – The first *N* frames (default 8) are processed together using bidirectional attention via a scale token to establish initial scene scale and depth estimates.

3. **KV-cache streaming** – Subsequent frames are processed causally. When `keyframe_interval=1`, every frame is cached; with `keyframe_interval>1`, only every *k*-th frame is stored. Non-keyframes trigger `_set_skip_append(True)` to discard KV after the forward pass, drastically reducing memory usage.

4. **Windowed mode** – For sequences exceeding the training range, the KV cache evicts old entries using a sliding window (`window_size`, `overlap_size`). The `inference_windowed()` method in [`benchmark/methods/lingbot_map.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/methods/lingbot_map.py) wraps this logic and handles flow-based keyframe selection.

5. **Post-processing** – Raw pose encodings (`pose_enc`) are decoded into extrinsic/intrinsic matrices using `pose_encoding_to_extri_intri`, depth maps are reshaped, and optional confidence maps are attached.

## Key Implementation Files

The following files define the core architecture of LingBot-Map:

- **[`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py)** – Contains the `GCTStream` class, `AggregatorStream`, `CameraCausalHead`, and the KV-cache management logic.

- **[`lingbot_map/models/gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py)** – Variant implementation supporting sliding-window KV eviction for ultra-long sequences.

- **[`benchmark/methods/lingbot_map.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/methods/lingbot_map.py)** – Benchmark adapter that exposes the model to evaluation pipelines, handling checkpoint loading and result formatting.

- **[`demo_render/batch_demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo_render/batch_demo.py)** – End-to-end offline rendering pipeline demonstrating the streaming engine on video sequences.

- **[`README.md`](https://github.com/Robbyant/lingbot-map/blob/main/README.md)** – High-level documentation and quick-start commands.

## Practical Usage Examples

### Interactive Streaming Demo

Run the model on an image sequence with adjustable keyframe intervals to control memory usage:

```bash
python demo.py \
    --model_path /path/to/lingbot-map-long.pt \
    --image_folder example/oxford \
    --mask_sky \
    --keyframe_interval 2

```

This creates a `LingbotMapMethod` in streaming mode and calls `inference_streaming()`, caching only every second frame.

### Programmatic Inference

Instantiate the `GCTStream` class directly for custom pipelines:

```python
import torch
from lingbot_map.models.gct_stream import GCTStream

model = GCTStream(
    img_size=518,
    patch_size=14,
    enable_3d_rope=True,
    kv_cache_sliding_window=64,
    kv_cache_scale_frames=8,
    use_sdpa=False,  # FlashInfer backend

)

ckpt = torch.load("lingbot-map-long.pt", map_location="cpu")
model.load_state_dict(ckpt["model"], strict=False)
model.eval().cuda()

# Prepare a list of RGB frames (HxWx3 uint8)

rgb_list = [...]  # e.g. from a video decoder

imgs = torch.stack([torchvision.transforms.ToTensor()(im) for im in rgb_list]).cuda()

preds = model.inference_streaming(
    imgs,
    num_scale_frames=8,
    keyframe_interval=4,
    output_device=torch.device("cpu")
)

```

### Windowed Inference for Long Sequences

For videos exceeding 10,000 frames, use the windowed variant:

```python
from lingbot_map.models.gct_stream_window import GCTStream

model = GCTStream(
    img_size=518,
    patch_size=14,
    enable_3d_rope=True,
    kv_cache_sliding_window=64,
    kv_cache_scale_frames=8,
    use_sdpa=False,
)

preds = model.inference_windowed(
    imgs,
    window_size=128,
    overlap_size=8,
    keyframe_interval=13,
    flow_threshold=0.0,
    max_non_keyframe_gap=30,
)

```

This automatically resets the KV cache after each window, enabling stable inference beyond the 320-frame RoPE training limit.

### Benchmark Adapter Integration

Use the benchmark wrapper for standardized evaluation:

```python
from benchmark.methods.lingbot_map import LingbotMapMethod

method = LingbotMapMethod(
    checkpoint="lingbot-map-long.pt",
    device="cuda",
    mode="streaming",
    keyframe_interval="auto",
)

# `gt_artifact` is a benchmark dataset object providing `load_rgb_list()`

results = method.process_scene(gt_artifact)

```

The `process_scene()` method internally calls `_prepare_images()`, `_run_inference()`, and `_process_outputs()` to return evaluation-compatible results.

### Offline Rendering Pipeline

Process long videos end-to-end with the rendering script:

```bash
python demo_render/batch_demo.py \
    --video_path /data/demo_videos/indoor_travel.MP4 \
    --output_folder /data/outputs/indoor_travel/ \
    --model_path /path/to/lingbot-map.pt \
    --config demo_render/config/indoor.yaml \
    --mode windowed --window_size 128 \
    --keyframe_interval 13 --overlap_keyframes 8 \
    --mask_sky \
    --save_predictions

```

This script writes an MP4 fly-through with optional sky-mask visualizations.

## Summary

- LingBot-Map implements a **Geometric Context Transformer (GCT)** combining an `AggregatorStream`, `CameraCausalHead`, and `GCTStream` engine.
- **KV-cache management** with configurable `keyframe_interval` bounds memory usage by caching only essential frames.
- **3-D Rotary Positional Encoding (RoPE)** maintains geometric consistency across the temporal dimension.
- **Dual inference modes**—streaming for real-time processing and windowed for sequences exceeding 10,000 frames—provide flexibility for diverse deployment scenarios.
- All core functionality resides in [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py) and is exposed via the `LingbotMapMethod` benchmark adapter.

## Frequently Asked Questions

### What is the Geometric Context Transformer in LingBot-Map?

The Geometric Context Transformer (GCT) is the central neural architecture that fuses visual features with 3-D positional information through causal attention mechanisms. It processes video streams frame-by-frame rather than requiring complete sequences upfront, enabling the real-time 3D reconstruction capabilities that define the core architecture of LingBot-Map.

### How does LingBot-Map prevent GPU memory overflow on long videos?

The system uses a **paged KV-cache** with a configurable `keyframe_interval` parameter that restricts caching to every *k*-th frame. Additionally, the `inference_windowed()` method implements sliding-window eviction (`window_size` and `overlap_size`) to reset the cache periodically, allowing processing of sequences that far exceed the 320-frame training limit.

### What distinguishes the streaming mode from the windowed inference mode?

**Streaming mode** processes frames sequentially with a monotonically growing KV-cache (limited by keyframe sampling), ideal for real-time applications. **Windowed mode** divides sequences into overlapping chunks, completely resetting the KV cache between windows to handle arbitrarily long videos while maintaining geometric consistency through shared keyframes.

### Where is the camera pose refinement logic implemented?

Camera pose refinement resides in the `CameraCausalHead` class, created by `_build_camera_head()` in [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py). This head receives aggregated tokens from the KV-cache and applies causal attention with optional 3-D RoPE to iteratively refine extrinsic and intrinsic camera parameters using the `camera_num_iterations` setting.