# How the Geometric Context Transformer Handles Streaming 3D Reconstruction: A Deep Dive into the GCTStream Architecture

> Unlock real-time streaming 3D reconstruction with the Geometric Context Transformer. Learn how GCTStream uses temporal causal attention and KV-caching for efficient, continuous processing.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: deep-dive
- Published: 2026-07-29

---

**The Geometric Context Transformer (GCT) enables real-time streaming 3D reconstruction through temporal causal attention, KV-cache persistence, and 3-D Rotary Positional Embeddings that process each new frame in constant time without reprocessing historical data.**

The **Geometric Context Transformer** is a vision-transformer-based model that typically operates on batches of RGB frames to estimate camera pose, depth, and dense 3D points. When adapted for **streaming 3D reconstruction**, the architecture introduces causal attention mechanisms and memory-efficient caching strategies that allow it to incrementally build scene geometry as new frames arrive. According to the `lingbot-map` repository, this streaming capability is implemented through specialized subclasses that extend the base GCT with online processing capabilities.

## Streaming Architecture Overview

The streaming variant of GCT transforms the standard bidirectional transformer into a causal, memory-bounded system suitable for real-time applications. Unlike batch processing that requires the entire video sequence upfront, the streaming architecture maintains a **KV-cache (key-value cache)** of attention statistics from previous frames, enabling *O(1)* processing time per new frame.

The system combines four critical mechanisms to maintain temporal consistency while controlling memory growth:

- **Temporal causal attention** restricts each frame to attend only to past frames, never future ones
- **Sliding-window KV-cache** evicts old frames after a configurable threshold (e.g., 64 frames)
- **Scale-token initialization** processes initial frames bidirectionally to establish metric scale
- **3-D Rotary Positional Embedding (3-D RoPE)** injects temporal position information into attention layers

## Key Components of the Streaming Pipeline

### Temporal Causal Attention and KV-Cache

At the heart of the streaming capability lies the **KV-cache**, implemented in `AggregatorStream` ([`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py)). This cache stores key and value tensors from previous attention computations, allowing the model to compute attention for new tokens by only calculating query-key interactions with the cached history rather than reprocessing entire sequences.

The cache supports two backends:
- **FlashInfer manager** for paged GPU cache when `use_sdpa=False`
- **Python dictionary** for standard PyTorch SDPA (Scaled Dot Product Attention)

The `_process_causal_stream` method (`lingbot_map/aggregator/stream.py#L514-L560`) handles the core logic, retrieving cached KV pairs and appending new frame data while respecting the causal mask.

### 3-D Rotary Positional Embedding

To maintain temporal coherence across streaming frames, the architecture implements **3-D RoPE** through the `WanRotaryPosEmbed` class ([`lingbot_map/layers/rope.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/rope.py)). Unlike standard 2D positional encodings, this mechanism generates complex rotation embeddings that encode the temporal index of each frame as a third dimension.

When `enable_3d_rope=True`, the `AggregatorStream` initializes this embedding in `_init_3d_rope` (`lingbot_map/aggregator/stream.py#L50-L64`) and applies it during `_process_causal_stream` to ensure that tokens from different timesteps receive appropriate positional signals even as the cache slides.

### Scale Frame Initialization

Before pure streaming begins, the model processes **scale frames** (typically the first 2-8 frames) using bidirectional attention. These frames can attend to each other via a special **scale token**, allowing the network to establish consistent metric scale for depth estimation.

In `GCTStream.inference_streaming` (`lingbot_map/models/gct_stream.py#L49-L102`), the method first processes `num_scale_frames` jointly, then transitions to causal mode for remaining frames. The `_prepare_special_tokens` method (`lingbot_map/aggregator/stream.py#L298-L368`) expands camera, register, and scale tokens to match the effective sequence length including cached frames.

## Core Implementation Classes

### GCTStream Orchestration Layer

The `GCTStream` class ([`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py)) subclasses `GCTBase` and serves as the primary interface for streaming inference. During initialization (`__init__`), it configures KV-cache parameters including `kv_cache_sliding_window` and `kv_cache_scale_frames` that control memory boundaries.

The `inference_streaming` method implements the core streaming loop:
1. Processes scale frames jointly with bidirectional attention
2. Iterates over remaining frames one-by-one with `causal_inference=True`
3. Implements **keyframe skipping** via `_set_skip_append` (`lingbot_map/models/gct_stream.py#L100-L119`), where non-keyframes set `_skip_append=True` to avoid polluting the cache
4. Offloads predictions to CPU via `output_device` to prevent GPU memory overflow

### AggregatorStream Causal Processing

`AggregatorStream` ([`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py)) extends `AggregatorBase` with streaming-specific attention. It maintains `total_frames_processed` to track the temporal index for RoPE calculations and manages the KV-cache lifecycle.

The class handles **sliding-window eviction** automatically: when the number of cached frames exceeds `kv_cache_sliding_window`, older frames are removed from the cache while preserving the scale frames (if configured via `kv_cache_scale_frames`). This ensures that long video sequences can be processed with bounded memory consumption.

### CameraCausalHead Pose Refinement

The `CameraCausalHead` ([`lingbot_map/heads/camera_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/camera_head.py)) mirrors the standard `CameraHead` but adds KV-cache persistence for camera pose estimation. It maintains `self.kv_cache` as a list of dictionaries, with one entry per refinement iteration.

Key features include:
- **Per-iteration caching**: Each step of the iterative refinement loop stores its own KV entries
- **3-D RoPE for camera tokens**: When enabled, `self.rope3d` generates positional embeddings specific to the camera token's temporal index
- **Frame tracking**: `self.frame_idx` increments after each call to ensure correct temporal positioning

The `trunk_fn` method (`lingbot_map/heads/camera_head.py#L96-L138`) implements iterative pose refinement, using cached attention when `causal_inference=True` and activating poses via `activate_pose` after each iteration.

## Generating 3D World Points from Streaming Data

After the forward pass generates depth maps and pose encodings for each frame, the system converts these into world-space coordinates through incremental unprojection. The `_unproject_depth_to_world` method in `GCTBase` (`lingbot_map/models/gct_base.py#L50-L86`) transforms per-pixel depth values and camera pose encodings into 3D coordinates.

Because this operation is applied frame-by-frame during the streaming loop, the resulting `world_points` tensor accumulates incrementally without requiring batch reprocessing. Each frame contributes its own slice to the global point cloud, enabling real-time visualization or incremental saving of the reconstruction.

## Code Example: Running Streaming Inference

```python
import torch
from lingbot_map.models.gct_stream import GCTStream

# 1️⃣ Instantiate the streaming model with KV-cache and 3-D RoPE

model = GCTStream(
    img_size=518,
    patch_size=14,
    embed_dim=1024,
    enable_3d_rope=True,                # Enable temporal consistency

    kv_cache_sliding_window=64,          # Evict frames after 64 steps

    kv_cache_scale_frames=8,             # Retain scale frames in cache

    enable_camera=True,
    enable_depth=True,
)

# Move to device (GPU required for FlashInfer, recommended for speed)

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model.to(device).eval()

# 2️⃣ Prepare video tensor: [S, 3, H, W] with values in [0, 1]

#    Here we simulate a 10-frame sequence

video = torch.rand(10, 3, 518, 518, device=device)

# 3️⃣ Execute streaming inference

#    - First 2 frames processed jointly as scale frames

#    - KV-cache updated only every 3rd frame (keyframe_interval)

outputs = model.inference_streaming(
    images=video,
    num_scale_frames=2,
    keyframe_interval=3,
    output_device=torch.device('cpu')  # Keep GPU memory bounded

)

# 4️⃣ Extract reconstruction results

pose_enc = outputs['pose_enc']           # [1, S, 9] camera poses

depth = outputs['depth']                 # [1, S, H, W, 1] depth maps

world_points = outputs['world_points']   # [1, S, H, W, 3] 3D coordinates

print(f"Streamed {pose_enc.shape[1]} frames, generated {world_points.numel()//3} 3D points")

```

This example demonstrates:
- **Memory management** through `kv_cache_sliding_window` and `output_device` offloading
- **Scale initialization** via `num_scale_frames=2` for accurate metric reconstruction
- **Selective caching** via `keyframe_interval=3` to reduce computation while maintaining prediction quality for every frame

## Summary

- **Temporal causal attention** in `AggregatorStream` restricts each frame to past context only, enabling real-time processing without future dependency.
- **KV-cache management** with sliding windows (configured via `kv_cache_sliding_window`) bounds memory usage to *O(window_size)* rather than sequence length.
- **3-D Rotary Positional Embedding** provides temporal positional encoding through `WanRotaryPosEmbed`, maintaining consistency across sliding windows.
- **Scale frame initialization** processes initial frames bidirectionally to establish metric scale before switching to causal streaming mode.
- **CameraCausalHead** maintains separate KV caches for pose refinement, updating incrementally via `trunk_fn` without recomputing historical poses.

## Frequently Asked Questions

### How does the KV-cache prevent memory overflow during long video streams?

The system implements a **sliding-window mechanism** configured via `kv_cache_sliding_window` (typically set to 64 frames). When the cache exceeds this limit, older frames are evicted while preserving scale frames (controlled by `kv_cache_scale_frames`). Additionally, **keyframe skipping** via `keyframe_interval` allows the model to process every frame for prediction while only caching KV tensors for selected keyframes, significantly reducing memory growth.

### What is the purpose of 3-D Rotary Positional Embedding in streaming mode?

**3-D RoPE** encodes the temporal index of each frame as a rotational transformation in the attention mechanism. Unlike standard positional encodings that require fixed sequence lengths, this approach from `WanRotaryPosEmbed` ([`lingbot_map/layers/rope.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/rope.py)) allows the model to distinguish between tokens from different timestamps even as the KV-cache slides and old frames are discarded. This maintains temporal coherence in the reconstruction without requiring absolute position indices.

### Why are scale frames processed differently from the rest of the stream?

The first `num_scale_frames` frames (typically 2-8) are processed with **bidirectional attention** using a special scale token. This allows these frames to attend to each other mutually, establishing consistent metric scale for depth estimation. Without this initialization step, the streaming causal model would lack sufficient context to determine absolute scene scale, as each subsequent frame can only attend to previous frames. After scale initialization, the model switches to causal-only processing for real-time operation.

### Can the streaming architecture handle variable frame rates or dropped frames?

Yes, the architecture handles temporal irregularities through the **frame index tracking** maintained in `CameraCausalHead.frame_idx` and `AggregatorStream.total_frames_processed`. Since 3-D RoPE generates positional embeddings based on the actual frame counter rather than sequence position, variable frame rates are naturally accommodated. If frames are dropped, the temporal indices remain consistent, ensuring that the positional embeddings correctly reflect the temporal gaps in the video stream.