Geometric Context Transformer in LingBot-Map: Architecture and Implementation Guide

The Geometric Context Transformer (GCT) is a Vision-Transformer-based architecture that processes video streams to output camera poses, dense depth maps, and 3D world points through a modular pipeline of patch embedding, transformer aggregation with optional KV-caching and 3D Rotary Positional Encoding (RoPE), and DPT-based prediction heads.

LingBot-Map builds its 3D scene-understanding backbone on the Geometric Context Transformer. This architecture transforms sequential RGB frames into coherent geometric contexts—camera poses, dense depth, and point clouds—using a three-stage pipeline designed for both accuracy and streaming efficiency. The implementation leverages DINOv2-derived patch embeddings, multi-scale transformer blocks, and specialized dense prediction heads to deliver real-time 3D geometry estimation.

Core Components of the Geometric Context Transformer

The Geometric Context Transformer architecture consists of three primary components orchestrated by the abstract GCTBase class. These modules handle tokenization, temporal feature aggregation, and geometric decoding.

Patch Embedding and Tokenization

The input pipeline begins with a patch embedding layer that converts raw RGB images into transformer-compatible token sequences.

• Implementation: lingbot_map/layers/patch_embed.py
• Default configuration: DINOv2 ViT-L/14 with registers (dinov2_vitl14_reg)
• Key parameters: img_size=518, patch_size=14, embed_dim=1024

This component slices each input frame into non-overlapping patches and projects them into a high-dimensional embedding space, producing a sequence of tokens that feed into the transformer aggregator.

Transformer Aggregator with Multi-Scale Processing

The aggregator forms the computational core of the Geometric Context Transformer, processing token sequences through multi-scale transformer blocks.

• Implementation: lingbot_map/aggregator/stream.py
• Key features:
  • 3D Rotary Positional Encoding (RoPE): Injects temporal sinusoids into attention scores when enable_3d_rope=True, maintaining geometric consistency across frames
  • KV-cache: Enables causal attention for streaming inference, where each frame attends only to cached tokens from previous frames via kv_cache_sliding_window parameters
  • FlashInfer integration: Optional optimized attention kernels when use_flashinfer=True

The aggregator handles both bidirectional processing for initialization frames and causal processing for streaming video, dramatically reducing GPU memory requirements during online inference.

Prediction Heads for Geometric Decoding

The aggregated tokens pass through task-specific prediction heads that decode features into concrete geometric outputs:

• Depth head: Located in lingbot_map/heads/dpt_head.py, this Dense Prediction Transformer (DPT) head outputs a 2-channel map containing depth values and confidence scores.
• Point head: Also implemented in dpt_head.py, but configured for 4-channel output (XYZ coordinates plus confidence), generating world-space point clouds.
• Camera head: Implemented in lingbot_map/heads/camera_head.py as a CameraCausalHead, this causal transformer refines 9-dimensional pose encodings (camera center plus quaternion) per frame using sliding window attention.

All heads share the same embedding dimension (2 * embed_dim) and operate on the patch-wise token stream produced by the aggregator.

The GCTBase Abstraction

The GCTBase class in lingbot_map/models/gct_base.py serves as the architectural scaffold, defining the common forward pipeline and hyper-parameter management.

The base class initializes with configuration flags for enabling specific outputs:

class GCTBase(nn.Module, PyTorchModelHubMixin, ABC):
    def __init__(self,
                 img_size: int = 518,
                 patch_size: int = 14,
                 embed_dim: int = 1024,
                 patch_embed: str = 'dinov2_vitl14_reg',
                 enable_camera: bool = True,
                 enable_point: bool = True,
                 enable_depth: bool = True,
                 enable_3d_rope: bool = False,
                 use_gradient_checkpoint: bool = True):
        # Builds aggregator and heads based on flags

        self.aggregator = self._build_aggregator()
        self.camera_head = self._build_camera_head() if enable_camera else None
        self.point_head = self._build_point_head() if enable_point else None
        self.depth_head = self._build_depth_head() if enable_depth else None

The forward implementation (lines 87-120 in gct_base.py) first calls _aggregate_features (implemented by concrete subclasses) and then executes the _predict_* helpers to generate the final prediction dictionary containing pose_enc, depth, world_points, and associated confidence maps.

Streaming Variants for Real-Time Processing

LingBot-Map provides two concrete implementations of the Geometric Context Transformer optimized for different inference scenarios.

GCTStream for Online Video Processing

GCTStream in lingbot_map/models/gct_stream.py inherits from GCTBase and specializes the aggregator for causal streaming:

class GCTStream(GCTBase):
    def _build_aggregator(self) -> nn.Module:
        return AggregatorStream(
            img_size=self.img_size,
            patch_size=self.patch_size,
            embed_dim=self.embed_dim,
            patch_embed=self.patch_embed,
            use_flashinfer=not self.use_sdpa,
            kv_cache_sliding_window=self.kv_cache_sliding_window,
            # ... additional parameters

        )

The inference_streaming method (lines 459-511) implements the following logic:

1. Normalizes input frames (shape (B, S, 3, H, W))
2. Cleans the KV cache for stateful processing
3. Processes initial scale frames with bidirectional attention
4. Streams subsequent frames with causal attention, optionally skipping KV-cache updates for non-keyframes based on keyframe_interval

GCTStreamWindow for Long-Sequence Handling

GCTStreamWindow in lingbot_map/models/gct_stream_window.py extends the streaming capability to arbitrarily long videos through windowed processing.

The inference_windowed method (lines 1120-1190) implements:

• Windowing: Splits long sequences into overlapping chunks (configured via window_size and overlap_size)
• Alignment: Uses _pairwise_alignment to compute learned similarity transforms between adjacent windows
• Stitching: Concatenates results via _stitch_windows to maintain a global coordinate frame

This approach allows the Geometric Context Transformer to process long videos while preserving global consistency, automatically managing KV cache resets between windows.

Implementation Examples

Instantiating a Streaming GCT Model

Configure the Geometric Context Transformer for temporal consistency and efficient inference:

from lingbot_map.models.gct_stream import GCTStream

model = GCTStream(
    img_size=518,
    patch_size=14,
    embed_dim=1024,
    patch_embed='dinov2_vitl14_reg',
    enable_camera=True,
    enable_depth=True,
    enable_point=False,
    enable_3d_rope=True,          # Enable temporal consistency

    sliding_window_size=64,       # Attention window size

    kv_cache_sliding_window=64,    # KV cache limit

    kv_cache_scale_frames=8,
)

model = model.cuda().eval()

Running Streaming Inference

Process video tensors with causal attention and keyframe subsampling:

import torch

# Input: (Batch, Sequence, Channels, Height, Width)

frames = torch.randn(1, 120, 3, 518, 518).cuda()

preds = model.inference_streaming(
    images=frames,
    num_scale_frames=1,
    keyframe_interval=4,
    output_device=torch.device('cpu')
)

# Extract predictions

pose_enc = preds['pose_enc']           # (B, S, 9)

depth = preds['depth']                 # (B, S, H, W, 1)

world_points = preds['world_points']   # (B, S, H, W, 3)

Processing Long Videos with Windowed Inference

Handle arbitrarily long sequences using the windowed variant:

preds = model.inference_windowed(
    images=frames,
    window_size=16,               # 16 keyframes per window

    overlap_size=8,               # 8 frame overlap

    keyframe_interval=2,
    flow_threshold=1.0,           # Flow-based keyframe selection

    output_device=torch.device('cpu')
)

Summary

The Geometric Context Transformer in LingBot-Map delivers accurate 3D scene understanding through a modular, efficient architecture:

• Three-stage pipeline: DINOv2 patch embedding, transformer aggregation with optional 3D RoPE, and DPT-based prediction heads
• Streaming optimization: KV-cache and causal attention in GCTStream enable real-time processing with constant memory usage
• Long-sequence support: GCTStreamWindow provides windowed inference with automatic alignment and stitching for arbitrary video lengths
• Flexible outputs: Configurable heads for camera pose (9-DOF), depth estimation, and world point cloud generation
• Source files: Core logic resides in lingbot_map/models/gct_base.py, gct_stream.py, and gct_stream_window.py, with components in lingbot_map/aggregator/stream.py and lingbot_map/heads/

Frequently Asked Questions

What is the difference between GCTStream and GCTStreamWindow?

GCTStream processes video sequentially using a KV-cache for causal attention, ideal for online streaming applications with fixed memory budgets. GCTStreamWindow extends this capability by splitting long videos into overlapping windows, aligning them via learned similarity transforms (_pairwise_alignment), and stitching results to maintain global geometric consistency across arbitrary sequence lengths.

How does the 3D Rotary Positional Encoding (RoPE) improve temporal consistency?

When enable_3d_rope=True, the Geometric Context Transformer injects temporal sinusoidal encodings into the attention mechanism. This 3D RoPE modifies attention scores based on relative temporal positions, helping the model maintain consistent geometric relationships across frames without requiring explicit temporal convolutions or recurrent states.

Why does the architecture use DINOv2 for patch embedding?

The patch embedding layer in lingbot_map/layers/patch_embed.py leverages DINOv2 ViT-L/14 with registers by default because DINOv2 provides robust, pre-trained visual features that capture both low-level textures and high-level semantic information. This initialization significantly improves convergence and final accuracy for downstream geometric prediction tasks compared to training from scratch.

What are the memory benefits of KV-caching in the Geometric Context Transformer?

The KV-cache implementation in AggregatorStream stores only the key and value tensors from previous frames rather than recomputing attention with the full history. During streaming inference (as implemented in GCTStream), this reduces memory complexity from quadratic in sequence length to constant (bounded by kv_cache_sliding_window), enabling the processing of infinite video streams on limited GPU memory.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}