How LingBot-Map's Geometric Context Transformer Works: Architecture and Implementation

The Geometric Context Transformer (GCT) is a Vision-Transformer-style network that converts RGB video streams into coherent 3D geometric context—including camera poses, dense depth maps, and world-coordinate point clouds—through a three-stage pipeline of patch embedding, transformer aggregation with optional KV-caching, and task-specific prediction heads.

LingBot-Map builds its scene-understanding backbone on the Geometric Context Transformer (GCT) architecture. This system processes temporal sequences of images to produce frame-wise geometric predictions suitable for SLAM, mapping, and AR/VR pipelines. The implementation resides in the Robbyant/lingbot-map repository, where the core logic is split between abstract base classes, streaming variants, and specialized prediction heads.

Core Architecture Components

The GCT architecture comprises three integrated components that transform raw pixels into structured 3D geometry.

Patch Embedding and Tokenization

The input pipeline begins in lingbot_map/layers/patch_embed.py, where a DINOv2-derived patch embedder converts each input image into a sequence of tokens. By default, the system uses patch_embed='dinov2_vitl14_reg' with a configuration of img_size=518, patch_size=14, and embed_dim=1024. This tokenization step produces a sequence of patch embeddings that serve as the input to the transformer aggregator.

Transformer Aggregation

The central processing engine lives in lingbot_map/aggregator/stream.py. The AggregatorStream class implements multi-scale transformer blocks that process the token sequence with two key optimizations:

• 3-D Rotary Positional Encoding (RoPE): When enable_3d_rope=True, the aggregator injects temporal sinusoids into attention scores to maintain geometric consistency across frames.
• KV-Cache for Streaming: The kv_cache_sliding_window parameter enables causal attention during inference, allowing each new frame to attend only to cached tokens from previous frames rather than recomputing the full sequence attention.

The aggregator supports use_flashinfer for optimized memory access and integrates with PyTorch's SDPA (Scaled Dot-Product Attention) when configured.

Prediction Heads

Located in lingbot_map/heads/, the output layers decode aggregated tokens into concrete geometric quantities:

• Depth Head (lingbot_map/heads/dpt_head.py): A Dense Prediction Transformer (DPT) head that outputs a 2-channel map containing depth and depth_confidence.
• Point Head (lingbot_map/heads/dpt_head.py): Another DPT variant outputting 4 channels (XYZ coordinates plus confidence) for world-coordinate point clouds.
• Camera Head (lingbot_map/heads/camera_head.py): A CameraCausalHead that implements causal transformer decoding to produce a 9-dimensional pose encoding (camera center plus quaternion) per frame.

All heads operate on an embedding dimension of 2 * embed_dim and share the patch-wise token stream produced by the aggregator.

The GCTBase Abstraction

The file lingbot_map/models/gct_base.py defines the GCTBase abstract class that serves as the architectural scaffold for all GCT variants. This class stores hyper-parameters—including img_size, patch_size, and embed_dim—and constructs the aggregator and heads through factory methods:

class GCTBase(nn.Module, PyTorchModelHubMixin, ABC):
    def __init__(self,
                 img_size: int = 518,
                 patch_size: int = 14,
                 embed_dim: int = 1024,
                 patch_embed: str = 'dinov2_vitl14_reg',
                 enable_camera: bool = True,
                 enable_point: bool = True,
                 enable_depth: bool = True,
                 enable_3d_rope: bool = False,
                 use_gradient_checkpoint: bool = True):
        ...
        self.aggregator = self._build_aggregator()
        self.camera_head = self._build_camera_head() if enable_camera else None
        self.point_head = self._build_point_head() if enable_point else None
        self.depth_head = self._build_depth_head() if enable_depth else None

The forward pipeline (lines 87-120 in gct_base.py) first calls _aggregate_features (implemented by subclasses) and then executes the _predict_* helpers to return a dictionary containing pose_enc, depth, depth_conf, world_points, and world_points_conf.

Streaming Inference Variants

For real-time applications, LingBot-Map provides two concrete implementations of GCTBase optimized for streaming video.

GCTStream

The GCTStream class in lingbot_map/models/gct_stream.py specializes the base class for causal, frame-by-frame processing. It constructs an AggregatorStream with KV-cache support:

class GCTStream(GCTBase):
    def _build_aggregator(self) -> nn.Module:
        return AggregatorStream(
            img_size=self.img_size,
            patch_size=self.patch_size,
            embed_dim=self.embed_dim,
            patch_embed=self.patch_embed,
            use_flashinfer=not self.use_sdpa,
            kv_cache_sliding_window=self.kv_cache_sliding_window,
            ...
        )

The inference_streaming method (lines 459-511) handles the end-to-end flow: it normalizes inputs, cleans the KV-cache, processes initial scale frames with bidirectional attention, then streams subsequent frames while optionally skipping KV-cache writes for non-keyframes based on the keyframe_interval parameter.

GCTStreamWindow

For arbitrarily long sequences, GCTStreamWindow in lingbot_map/models/gct_stream_window.py extends the streaming core with windowed inference. It splits video into overlapping windows, processes each with a fresh KV-cache, and aligns results via _pairwise_alignment (a learned similarity transform) before stitching them together with _stitch_windows (lines 1120-1190). This approach maintains a global coordinate frame while bounding memory usage.

End-to-End Inference Examples

Instantiating a Streaming Model

from lingbot_map.models.gct_stream import GCTStream

model = GCTStream(
    img_size=518,
    patch_size=14,
    embed_dim=1024,
    patch_embed='dinov2_vitl14_reg',
    enable_camera=True,
    enable_depth=True,
    enable_point=False,
    enable_3d_rope=True,          # Enable temporal consistency

    sliding_window_size=64,       # Attention window size

    kv_cache_sliding_window=64,   # Cache for causal attention

    kv_cache_scale_frames=8,
)
model = model.cuda().eval()

This configuration builds the aggregator and heads exactly as defined in lingbot_map/models/gct_stream.py (lines 31-70).

Running Streaming Inference

import torch

# Input: (B, S, 3, H, W) with values in [0, 1]

frames = torch.randn(1, 120, 3, 518, 518).cuda()

preds = model.inference_streaming(
    images=frames,
    num_scale_frames=1,
    keyframe_interval=4,
    output_device=torch.device('cpu')
)

# Extract predictions

pose_enc = preds['pose_enc']         # (B, S, 9)

depth = preds['depth']               # (B, S, H, W, 1)

world_points = preds['world_points'] # (B, S, H, W, 3)

The method internally manages the KV-cache state and applies 3-D RoPE to maintain temporal geometric consistency.

Processing Long Videos with Windowing

preds = model.inference_windowed(
    images=frames,
    window_size=16,               # Keyframes per window

    overlap_size=8,               # Overlapping frames between windows

    keyframe_interval=2,
    flow_threshold=1.0,           # Enable optical-flow keyframe selection

    output_device=torch.device('cpu')
)

This routine automatically handles window alignment and stitching, producing globally consistent predictions across the entire sequence.

Summary

• GCT processes RGB video through DINOv2 patch embedding, transformer aggregation with optional 3-D RoPE, and DPT-based prediction heads to output camera poses, depth, and 3D points.
• The GCTBase abstract class in lingbot_map/models/gct_base.py defines the common forward pipeline and head construction, while concrete subclasses handle specific inference modes.
• GCTStream enables real-time causal inference with KV-caching and sliding-window attention, drastically reducing GPU memory for long sequences.
• GCTStreamWindow extends streaming with automatic windowing, alignment (_pairwise_alignment), and stitching (_stitch_windows) to handle arbitrarily long videos while maintaining global geometric consistency.
• Prediction heads in lingbot_map/heads/ decode tokens into 9-D pose encodings (camera center + quaternion), 2-channel depth maps, and 4-channel world point clouds (XYZ + confidence).

Frequently Asked Questions

What is the primary purpose of the Geometric Context Transformer in LingBot-Map?

The Geometric Context Transformer serves as the core 3D scene-understanding engine that processes monocular video to simultaneously estimate camera ego-motion, per-pixel depth, and 3D world coordinates. According to the Robbyant/lingbot-map source code, it combines Vision-Transformer architectures with geometric inductive biases to produce outputs compatible with downstream SLAM and dense mapping systems.

How does the KV-cache improve streaming performance in GCT?

The KV-cache mechanism, implemented in lingbot_map/aggregator/stream.py and utilized by GCTStream, stores key and value tensors from previous frames during causal attention. This allows the model to process each new frame by attending only to the cached history rather than recomputing attention over the entire sequence, reducing GPU memory from quadratic to linear complexity with respect to sequence length.

What is the difference between GCTStream and GCTStreamWindow?

GCTStream processes video as a single continuous stream using a fixed-size sliding window and KV-cache, suitable for online robotics applications. GCTStreamWindow, defined in lingbot_map/models/gct_stream_window.py, automatically segments long videos into overlapping chunks, processes each chunk independently, and aligns them using _pairwise_alignment before stitching with _stitch_windows, making it ideal for offline batch processing of long recordings.

Which geometric outputs does the Geometric Context Transformer produce?

The transformer outputs four primary geometric quantities through its heads: pose_enc (9-dimensional camera pose encoding from lingbot_map/heads/camera_head.py), depth and depth_conf (metric depth with confidence from the DPT head), and world_points with world_points_conf (3D coordinates in global frame from the point head). Utilities in lingbot_map/utils/pose_enc.py and lingbot_map/utils/geometry.py convert these encodings into standard camera matrices and point clouds.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}