# Lingbot-Map Architecture: Inside the Geometric Context Transformer

> Discover the Lingbot-Map architecture's Geometric Context Transformer. Learn how it achieves real-time 3D reconstruction using multi-scale features, RoPE, and causal attention within limited GPU memory.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: architecture
- Published: 2026-07-19

---

**Lingbot-Map leverages a Geometric Context Transformer (GCT) that combines multi-scale visual feature aggregation, 3-D Rotary Positional Encoding (RoPE), and causal attention mechanisms to deliver real-time streaming 3-D reconstruction with bounded GPU memory.**

The lingbot-map architecture powers a state-of-the-art streaming reconstruction system available in the `Robbyant/lingbot-map` repository. Unlike batch-processing approaches that require entire video sequences upfront, this architecture processes frames causally using a novel KV-cached transformer design, enabling real-time performance on long video streams while maintaining geometric consistency through 3-D positional encodings.

## Core Architectural Components

The lingbot-map architecture consists of three tightly coupled modules that operate within a unified streaming framework defined in [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py).

### Feature Aggregator (AggregatorStream)

The `_build_aggregator()` method instantiates the `AggregatorStream` class, which extracts multi-scale visual tokens from each incoming frame. This component maintains tokens in a **paged KV-cache** and supports configurable parameters including `kv_cache_sliding_window` and `kv_cache_scale_frames`. It optionally applies **3-D RoPE** (Rotary Positional Encoding) to enforce temporal consistency across the token sequence, ensuring geometric stability as the camera moves through the scene.

### Camera Head (CameraCausalHead)

The `_build_camera_head()` method creates the `CameraCausalHead`, which refines per-frame camera poses using accumulated tokens from the aggregator. Operating strictly causally—attending only to past and current frames—this module supports iterative refinement via the `camera_num_iterations` parameter. It integrates with the keyframe logic to optimize pose estimation while managing computational overhead, and can utilize camera-specific 3-D RoPE configurations distinct from the visual branch.

### Streaming Engine (GCTStream)

The `GCTStream` class orchestrates the aggregator and camera head, implementing the main inference loop in `inference_streaming()`. This engine manages the complete KV-cache lifecycle, including cleaning operations, "skip-append" logic for non-keyframes via `_set_skip_append(True)`, and statistical tracking. It provides both **streaming** and **windowed** inference modes to handle variable sequence lengths efficiently without exhausting GPU memory.

## How the Streaming Pipeline Works

The architecture implements a five-phase processing pipeline that balances reconstruction accuracy with memory efficiency.

1. **Input Preprocessing**: Images are converted to tensor format `[S, 3, H, W]` via `LingbotMapMethod._prepare_images` in [`benchmark/methods/lingbot_map.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/methods/lingbot_map.py), with optional resizing to maintain consistent input dimensions.

2. **Scale Frame Initialization**: The first *N* frames (default 8) are processed together using bidirectional attention through a special scale token. This establishes the initial scene scale and provides baseline depth and pose estimates that anchor the subsequent causal processing.

3. **KV-Cache Streaming**: After initialization, subsequent frames are processed causally. When `keyframe_interval=1`, every frame's key-value pairs are cached. With `keyframe_interval > 1`, only every *k*-th frame is stored; non-keyframes trigger `_set_skip_append(True)` to discard their KV tensors immediately after the forward pass, drastically reducing memory consumption during long sequences.

4. **Windowed Mode for Long Sequences**: For sequences exceeding the 320-frame RoPE training range, the `inference_windowed()` method (available in both [`gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream.py) and [`gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream_window.py)) implements sliding-window cache eviction using `window_size` and `overlap_size` parameters. The benchmark adapter wraps this functionality with flow-based keyframe selection to maintain continuity across window boundaries.

5. **Post-Processing**: Raw pose encodings (`pose_enc`) are decoded into extrinsic and intrinsic camera matrices via `pose_encoding_to_extri_intri()` in [`lingbot_map/utils/pose_enc.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/pose_enc.py), while depth maps are reshaped to match input resolutions and optional confidence maps are attached to the output dictionary.

## Key Implementation Files

Understanding the lingbot-map architecture requires familiarity with these critical source files:

- **[`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py)**: Contains the core `GCTStream` class, `AggregatorStream`, `CameraCausalHead`, and the KV-cache management logic including the keyframe interval mechanisms.
- **[`lingbot_map/models/gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py)**: Specialized variant implementing sliding-window KV eviction for ultra-long sequences that exceed standard positional encoding limits.
- **[`benchmark/methods/lingbot_map.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/methods/lingbot_map.py)**: Benchmark adapter that handles checkpoint loading, image preparation via `_prepare_images()`, inference routing through `_run_inference()`, and result formatting via `_process_outputs()`.
- **[`demo_render/batch_demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo_render/batch_demo.py)**: End-to-end offline rendering pipeline demonstrating the streaming engine on long video sequences with configurable visualization options.

## Practical Usage Examples

### Running Streaming Inference

Execute real-time reconstruction with configurable keyframe intervals to control memory usage:

```bash
python demo.py \
    --model_path /path/to/lingbot-map-long.pt \
    --image_folder example/oxford \
    --mask_sky \
    --keyframe_interval 2

```

This command creates a `LingbotMapMethod` instance in streaming mode, where the KV-cache grows only on keyframes, maintaining bounded GPU memory usage suitable for real-time applications.

### Programmatic Model Initialization

Instantiate the architecture directly for custom pipelines:

```python
import torch
from lingbot_map.models.gct_stream import GCTStream

# Initialize streaming model with 3-D RoPE enabled

model = GCTStream(
    img_size=518,
    patch_size=14,
    enable_3d_rope=True,
    kv_cache_sliding_window=64,
    kv_cache_scale_frames=8,
    use_sdpa=False,  # Uses FlashInfer backend for fast KV paging

)

# Load checkpoint

ckpt = torch.load("lingbot-map-long.pt", map_location="cpu")
model.load_state_dict(ckpt["model"], strict=False)
model.eval().cuda()

# Process RGB frames

rgb_list = [...]  # List of HxWx3 uint8 images

imgs = torch.stack([torchvision.transforms.ToTensor()(im) for im in rgb_list]).cuda()

# Run inference with keyframe every 4 frames

preds = model.inference_streaming(
    imgs,
    num_scale_frames=8,
    keyframe_interval=4,
    output_device=torch.device("cpu")
)

# Decode camera parameters

from lingbot_map.utils.pose_enc import pose_encoding_to_extri_intri
extr, intr = pose_encoding_to_extri_intri(preds["pose_enc"], imgs.shape[-2:])

```

### Windowed Inference for Extended Sequences

Handle videos exceeding 10,000 frames using sliding windows to prevent cache overflow:

```python
from lingbot_map.models.gct_stream_window import GCTStream

model = GCTStream(
    img_size=518,
    patch_size=14,
    enable_3d_rope=True,
    kv_cache_sliding_window=64,
    kv_cache_scale_frames=8,
    use_sdpa=False,
)

# Windowed processing with automatic KV reset between windows

preds = model.inference_windowed(
    imgs,
    window_size=128,      # KV slots per window

    overlap_size=8,       # Shared keyframes between windows

    keyframe_interval=13,
    flow_threshold=0.0,
    max_non_keyframe_gap=30,
)

```

### Benchmark Integration

Integrate with evaluation frameworks using the provided adapter:

```python
from benchmark.methods.lingbot_map import LingbotMapMethod

method = LingbotMapMethod(
    checkpoint="lingbot-map-long.pt",
    device="cuda",
    mode="streaming",
    keyframe_interval="auto",  # Auto-select based on sequence length

)

results = method.process_scene(gt_artifact)

```

## Summary

- The lingbot-map architecture centers on a **Geometric Context Transformer (GCT)** that enables real-time streaming 3-D reconstruction through causal attention mechanisms.
- **Three core components**—the `AggregatorStream`, `CameraCausalHead`, and `GCTStream` engine—work together in [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py) to process visual tokens and camera poses sequentially.
- **KV-cache management** with configurable `keyframe_interval` parameters and the `_set_skip_append()` method ensures GPU memory remains bounded during long sequences.
- **Dual inference modes** (`inference_streaming()` and `inference_windowed()`) support both low-latency real-time applications and offline processing of ultra-long videos exceeding the 320-frame RoPE training range.
- **3-D Rotary Positional Encoding** maintains geometric consistency across the temporal dimension, distinguishing this architecture from standard 2-D vision transformers.

## Frequently Asked Questions

### What is the Geometric Context Transformer in lingbot-map?

The Geometric Context Transformer (GCT) is the core neural architecture that fuses multi-scale visual features with 3-D geometric information. It employs causal attention mechanisms and 3-D Rotary Positional Encoding (RoPE) to process video streams frame-by-frame while maintaining temporal consistency, as implemented in the `GCTStream` class within [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py). This design allows the model to understand spatial relationships in dynamic scenes without requiring future frame information.

### How does lingbot-map handle memory constraints during long video processing?

The architecture implements a **paged KV-cache** system with configurable eviction policies. Through the `keyframe_interval` parameter in `inference_streaming()`, non-keyframes can skip cache appending via `_set_skip_append(True)`, reducing cache growth. For extremely long sequences, the `inference_windowed()` method in [`gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream_window.py) provides sliding-window cache eviction with `window_size` and `overlap_size` controls, enabling processing of sequences that far exceed the model's 320-frame positional encoding limit.

### What distinguishes the streaming mode from windowed inference?

**Streaming mode** (`inference_streaming()`) processes frames continuously with a monotonically growing KV-cache (constrained only by keyframe skipping), making it ideal for real-time applications requiring immediate output. **Windowed inference** (`inference_windowed()`) periodically resets the KV-cache using overlapping windows, enabling stable processing of hour-long videos that would otherwise exceed the model's positional encoding range or available GPU memory.

### Where is the camera pose estimation logic located?

Camera pose refinement occurs in the `CameraCausalHead` class, constructed via `_build_camera_head()` in [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py). This module processes tokens from the `AggregatorStream` using causal self-attention and iterative refinement controlled by `camera_num_iterations`. Final pose decoding from latent encodings to extrinsic/intrinsic matrices is handled by `pose_encoding_to_extri_intri()` in [`lingbot_map/utils/pose_enc.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/pose_enc.py).