How the Geometric Context Transformer Handles Streaming 3D Reconstruction: A Deep Dive into the GCTStream Architecture

The Geometric Context Transformer (GCT) enables real-time streaming 3D reconstruction through temporal causal attention, KV-cache persistence, and 3-D Rotary Positional Embeddings that process each new frame in constant time without reprocessing historical data.

The Geometric Context Transformer is a vision-transformer-based model that typically operates on batches of RGB frames to estimate camera pose, depth, and dense 3D points. When adapted for streaming 3D reconstruction, the architecture introduces causal attention mechanisms and memory-efficient caching strategies that allow it to incrementally build scene geometry as new frames arrive. According to the lingbot-map repository, this streaming capability is implemented through specialized subclasses that extend the base GCT with online processing capabilities.

Streaming Architecture Overview

The streaming variant of GCT transforms the standard bidirectional transformer into a causal, memory-bounded system suitable for real-time applications. Unlike batch processing that requires the entire video sequence upfront, the streaming architecture maintains a KV-cache (key-value cache) of attention statistics from previous frames, enabling O(1) processing time per new frame.

The system combines four critical mechanisms to maintain temporal consistency while controlling memory growth:

• Temporal causal attention restricts each frame to attend only to past frames, never future ones
• Sliding-window KV-cache evicts old frames after a configurable threshold (e.g., 64 frames)
• Scale-token initialization processes initial frames bidirectionally to establish metric scale
• 3-D Rotary Positional Embedding (3-D RoPE) injects temporal position information into attention layers

Key Components of the Streaming Pipeline

Temporal Causal Attention and KV-Cache

At the heart of the streaming capability lies the KV-cache, implemented in AggregatorStream (lingbot_map/aggregator/stream.py). This cache stores key and value tensors from previous attention computations, allowing the model to compute attention for new tokens by only calculating query-key interactions with the cached history rather than reprocessing entire sequences.

The cache supports two backends:

• FlashInfer manager for paged GPU cache when use_sdpa=False
• Python dictionary for standard PyTorch SDPA (Scaled Dot Product Attention)

The _process_causal_stream method (lingbot_map/aggregator/stream.py#L514-L560) handles the core logic, retrieving cached KV pairs and appending new frame data while respecting the causal mask.

3-D Rotary Positional Embedding

To maintain temporal coherence across streaming frames, the architecture implements 3-D RoPE through the WanRotaryPosEmbed class (lingbot_map/layers/rope.py). Unlike standard 2D positional encodings, this mechanism generates complex rotation embeddings that encode the temporal index of each frame as a third dimension.

When enable_3d_rope=True, the AggregatorStream initializes this embedding in _init_3d_rope (lingbot_map/aggregator/stream.py#L50-L64) and applies it during _process_causal_stream to ensure that tokens from different timesteps receive appropriate positional signals even as the cache slides.

Scale Frame Initialization

Before pure streaming begins, the model processes scale frames (typically the first 2-8 frames) using bidirectional attention. These frames can attend to each other via a special scale token, allowing the network to establish consistent metric scale for depth estimation.

In GCTStream.inference_streaming (lingbot_map/models/gct_stream.py#L49-L102), the method first processes num_scale_frames jointly, then transitions to causal mode for remaining frames. The _prepare_special_tokens method (lingbot_map/aggregator/stream.py#L298-L368) expands camera, register, and scale tokens to match the effective sequence length including cached frames.

Core Implementation Classes

GCTStream Orchestration Layer

The GCTStream class (lingbot_map/models/gct_stream.py) subclasses GCTBase and serves as the primary interface for streaming inference. During initialization (__init__), it configures KV-cache parameters including kv_cache_sliding_window and kv_cache_scale_frames that control memory boundaries.

The inference_streaming method implements the core streaming loop:

1. Processes scale frames jointly with bidirectional attention
2. Iterates over remaining frames one-by-one with causal_inference=True
3. Implements keyframe skipping via _set_skip_append (lingbot_map/models/gct_stream.py#L100-L119), where non-keyframes set _skip_append=True to avoid polluting the cache
4. Offloads predictions to CPU via output_device to prevent GPU memory overflow

AggregatorStream Causal Processing

AggregatorStream (lingbot_map/aggregator/stream.py) extends AggregatorBase with streaming-specific attention. It maintains total_frames_processed to track the temporal index for RoPE calculations and manages the KV-cache lifecycle.

The class handles sliding-window eviction automatically: when the number of cached frames exceeds kv_cache_sliding_window, older frames are removed from the cache while preserving the scale frames (if configured via kv_cache_scale_frames). This ensures that long video sequences can be processed with bounded memory consumption.

CameraCausalHead Pose Refinement

The CameraCausalHead (lingbot_map/heads/camera_head.py) mirrors the standard CameraHead but adds KV-cache persistence for camera pose estimation. It maintains self.kv_cache as a list of dictionaries, with one entry per refinement iteration.

Key features include:

• Per-iteration caching: Each step of the iterative refinement loop stores its own KV entries
• 3-D RoPE for camera tokens: When enabled, self.rope3d generates positional embeddings specific to the camera token's temporal index
• Frame tracking: self.frame_idx increments after each call to ensure correct temporal positioning

The trunk_fn method (lingbot_map/heads/camera_head.py#L96-L138) implements iterative pose refinement, using cached attention when causal_inference=True and activating poses via activate_pose after each iteration.

Generating 3D World Points from Streaming Data

After the forward pass generates depth maps and pose encodings for each frame, the system converts these into world-space coordinates through incremental unprojection. The _unproject_depth_to_world method in GCTBase (lingbot_map/models/gct_base.py#L50-L86) transforms per-pixel depth values and camera pose encodings into 3D coordinates.

Because this operation is applied frame-by-frame during the streaming loop, the resulting world_points tensor accumulates incrementally without requiring batch reprocessing. Each frame contributes its own slice to the global point cloud, enabling real-time visualization or incremental saving of the reconstruction.

Code Example: Running Streaming Inference

import torch
from lingbot_map.models.gct_stream import GCTStream

# 1️⃣ Instantiate the streaming model with KV-cache and 3-D RoPE

model = GCTStream(
    img_size=518,
    patch_size=14,
    embed_dim=1024,
    enable_3d_rope=True,                # Enable temporal consistency

    kv_cache_sliding_window=64,          # Evict frames after 64 steps

    kv_cache_scale_frames=8,             # Retain scale frames in cache

    enable_camera=True,
    enable_depth=True,
)

# Move to device (GPU required for FlashInfer, recommended for speed)

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model.to(device).eval()

# 2️⃣ Prepare video tensor: [S, 3, H, W] with values in [0, 1]

#    Here we simulate a 10-frame sequence

video = torch.rand(10, 3, 518, 518, device=device)

# 3️⃣ Execute streaming inference

#    - First 2 frames processed jointly as scale frames

#    - KV-cache updated only every 3rd frame (keyframe_interval)

outputs = model.inference_streaming(
    images=video,
    num_scale_frames=2,
    keyframe_interval=3,
    output_device=torch.device('cpu')  # Keep GPU memory bounded

)

# 4️⃣ Extract reconstruction results

pose_enc = outputs['pose_enc']           # [1, S, 9] camera poses

depth = outputs['depth']                 # [1, S, H, W, 1] depth maps

world_points = outputs['world_points']   # [1, S, H, W, 3] 3D coordinates

print(f"Streamed {pose_enc.shape[1]} frames, generated {world_points.numel()//3} 3D points")

This example demonstrates:

• Memory management through kv_cache_sliding_window and output_device offloading
• Scale initialization via num_scale_frames=2 for accurate metric reconstruction
• Selective caching via keyframe_interval=3 to reduce computation while maintaining prediction quality for every frame

Summary

• Temporal causal attention in AggregatorStream restricts each frame to past context only, enabling real-time processing without future dependency.
• KV-cache management with sliding windows (configured via kv_cache_sliding_window) bounds memory usage to O(window_size) rather than sequence length.
• 3-D Rotary Positional Embedding provides temporal positional encoding through WanRotaryPosEmbed, maintaining consistency across sliding windows.
• Scale frame initialization processes initial frames bidirectionally to establish metric scale before switching to causal streaming mode.
• CameraCausalHead maintains separate KV caches for pose refinement, updating incrementally via trunk_fn without recomputing historical poses.

Frequently Asked Questions

How does the KV-cache prevent memory overflow during long video streams?

The system implements a sliding-window mechanism configured via kv_cache_sliding_window (typically set to 64 frames). When the cache exceeds this limit, older frames are evicted while preserving scale frames (controlled by kv_cache_scale_frames). Additionally, keyframe skipping via keyframe_interval allows the model to process every frame for prediction while only caching KV tensors for selected keyframes, significantly reducing memory growth.

What is the purpose of 3-D Rotary Positional Embedding in streaming mode?

3-D RoPE encodes the temporal index of each frame as a rotational transformation in the attention mechanism. Unlike standard positional encodings that require fixed sequence lengths, this approach from WanRotaryPosEmbed (lingbot_map/layers/rope.py) allows the model to distinguish between tokens from different timestamps even as the KV-cache slides and old frames are discarded. This maintains temporal coherence in the reconstruction without requiring absolute position indices.

Why are scale frames processed differently from the rest of the stream?

The first num_scale_frames frames (typically 2-8) are processed with bidirectional attention using a special scale token. This allows these frames to attend to each other mutually, establishing consistent metric scale for depth estimation. Without this initialization step, the streaming causal model would lack sufficient context to determine absolute scene scale, as each subsequent frame can only attend to previous frames. After scale initialization, the model switches to causal-only processing for real-time operation.

Can the streaming architecture handle variable frame rates or dropped frames?

Yes, the architecture handles temporal irregularities through the frame index tracking maintained in CameraCausalHead.frame_idx and AggregatorStream.total_frames_processed. Since 3-D RoPE generates positional embeddings based on the actual frame counter rather than sequence position, variable frame rates are naturally accommodated. If frames are dropped, the temporal indices remain consistent, ensuring that the positional embeddings correctly reflect the temporal gaps in the video stream.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}