Geometric Context Transformer Architecture in LingBot-Map: A Technical Deep Dive

The Geometric Context Transformer (GCT) is a Vision-Transformer-style network that processes RGB video streams to jointly estimate camera poses, dense depth maps, and 3D point clouds through patch embedding, streaming transformer aggregation with KV-caching, and specialized DPT-based prediction heads.

The Geometric Context Transformer serves as the core 3D scene-understanding backbone in the LingBot-Map repository. This architecture combines a DINOv2-derived patch embedder with causal transformer blocks to generate coherent geometric context from video frames. Written in PyTorch, the implementation provides both real-time streaming and windowed inference modes for handling arbitrarily long sequences.

Core Components of the Geometric Context Transformer

The GCT architecture consists of three interconnected stages orchestrated by the abstract GCTBase class. Each component handles a specific aspect of transforming raw pixels into structured 3D geometry.

Patch Embedding and Tokenization

The input pipeline begins in lingbot_map/layers/patch_embed.py, where a DINOv2-derived patch embedder converts each RGB frame into a sequence of tokens. By default, the model uses patch_embed='dinov2_vitl14_reg' with an image size of 518 pixels and a patch size of 14, producing tokens with an embedding dimension of 1024.

This tokenization process allows the transformer to process visual information using self-attention mechanisms, similar to how Vision Transformers handle image classification tasks.

Transformer Aggregator with KV-Cache

The central processing unit resides in lingbot_map/aggregator/stream.py within the AggregatorStream class. This component stacks multi-scale transformer blocks that process the token sequence temporally.

Key features of the aggregator include:

• KV-Cache Mechanism: Enables causal attention during streaming inference, where each new frame only attends to cached tokens from previous frames rather than recomputing the full sequence attention. This dramatically reduces GPU memory consumption during real-time processing.
• 3-D Rotary Positional Encoding (RoPE): When enable_3d_rope=True, the architecture injects temporal sinusoids into attention scores to maintain geometric consistency across time.
• Sliding Window Management: The kv_cache_sliding_window and kv_cache_scale_frames parameters control memory retention for long sequences.

Prediction Heads for Geometry

The aggregated tokens flow into three specialized heads located in lingbot_map/heads/:

• Depth Head (dpt_head.py): A Dense Prediction Transformer (DPT) head that outputs a 2-channel map containing depth values and confidence scores.
• Point Head (dpt_head.py): Another DPT head with 4 output channels (XYZ coordinates plus confidence) for generating world-space point clouds.
• Camera Head (camera_head.py): A CameraCausalHead that performs iterative refinement on a 9-dimensional pose encoding (camera center plus quaternion) using causal transformer blocks.

All heads operate on a shared embedding dimension of 2 * embed_dim and consume the patch-wise token stream produced by the aggregator.

GCTBase: The Abstract Scaffold

The GCTBase class in lingbot_map/models/gct_base.py defines the shared interface and initialization logic for all GCT variants. It handles hyper-parameter storage and head construction while leaving the aggregation implementation to subclasses.

class GCTBase(nn.Module, PyTorchModelHubMixin, ABC):
    def __init__(self,
                 img_size: int = 518,
                 patch_size: int = 14,
                 embed_dim: int = 1024,
                 patch_embed: str = 'dinov2_vitl14_reg',
                 enable_camera: bool = True,
                 enable_point: bool = True,
                 enable_depth: bool = True,
                 enable_3d_rope: bool = False,
                 use_gradient_checkpoint: bool = True):
        super().__init__()
        self.aggregator = self._build_aggregator()
        self.camera_head = self._build_camera_head() if enable_camera else None
        self.point_head = self._build_point_head() if enable_point else None
        self.depth_head = self._build_depth_head() if enable_depth else None

The forward pass (lines 87-120 in gct_base.py) first calls _aggregate_features (implemented by concrete subclasses) and then runs the four _predict_* helpers to generate the final prediction dictionary containing pose encodings, depth maps, and world points.

Streaming Variants for Real-Time Inference

LingBot-Map provides two concrete implementations of GCTBase optimized for different inference scenarios.

GCTStream for Causal Processing

GCTStream in lingbot_map/models/gct_stream.py specializes the base class for online video processing. It constructs an AggregatorStream with support for KV-caching and optional Flash Infer attention:

class GCTStream(GCTBase):
    def _build_aggregator(self) -> nn.Module:
        return AggregatorStream(
            img_size=self.img_size,
            patch_size=self.patch_size,
            embed_dim=self.embed_dim,
            patch_embed=self.patch_embed,
            pretrained_path=self.pretrained_path,
            use_flashinfer=not self.use_sdpa,
            kv_cache_sliding_window=self.kv_cache_sliding_window,
            enable_3d_rope=self.enable_3d_rope
        )

The causal attention mechanism ensures that each frame only attends to previous tokens, making the model suitable for real-time SLAM and odometry applications.

GCTStreamWindow for Long Sequences

GCTStreamWindow (located in lingbot_map/models/gct_stream_window.py) extends the streaming capability to arbitrarily long videos by processing overlapping windows. It splits sequences into manageable chunks, aligns them via _pairwise_alignment, and stitches results using _stitch_windows. This approach maintains a global coordinate frame while circumventing memory limitations on long videos.

End-to-End Inference Pipeline

Instantiating a Streaming Model

To create a GCT model with temporal consistency enabled:

from lingbot_map.models.gct_stream import GCTStream

model = GCTStream(
    img_size=518,
    patch_size=14,
    embed_dim=1024,
    patch_embed='dinov2_vitl14_reg',
    enable_camera=True,
    enable_depth=True,
    enable_point=False,
    enable_3d_rope=True,          # Enable temporal consistency

    sliding_window_size=64,        # KV cache window size

    kv_cache_sliding_window=64,
    kv_cache_scale_frames=8
).cuda().eval()

The constructor chains together the patch embedder, aggregator, and prediction heads exactly as defined in gct_stream.py (lines 31-70).

Running Streaming Inference

Process video sequences with causal KV-caching:

import torch

# frames shape: (Batch, Sequence, Channels, Height, Width)

frames = torch.randn(1, 120, 3, 518, 518).cuda()

preds = model.inference_streaming(
    images=frames,
    num_scale_frames=1,
    keyframe_interval=4,
    output_device=torch.device('cpu')
)

# Extract predictions

pose_enc = preds['pose_enc']        # (B, S, 9) - camera centers and quaternions

depth = preds['depth']              # (B, S, H, W, 1)

world_points = preds['world_points'] # (B, S, H, W, 3)

The inference_streaming method (lines 459-511 in gct_stream.py) handles input normalization, KV-cache initialization, and keyframe selection logic. It processes scale frames with bidirectional attention before switching to causal streaming mode.

Processing Long Videos with Windowed Inference

For sequences exceeding GPU memory limits:

preds = model.inference_windowed(
    images=frames,
    window_size=16,               # 16 keyframes per window

    overlap_size=8,               # 8 frames of overlap

    keyframe_interval=2,
    flow_threshold=1.0,           # Enable flow-based keyframe selection

    output_device=torch.device('cpu')
)

The windowed routine (lines 1120-1190 in gct_stream_window.py) automatically manages window boundaries, runs each segment with a fresh KV cache, and performs global alignment across windows.

Summary

• Geometric Context Transformer combines Vision-Transformer architecture with 3D geometric prediction heads for simultaneous camera pose estimation, depth prediction, and point cloud generation.
• Three-stage pipeline: Patch embedding (patch_embed.py), transformer aggregation with KV-cache (stream.py), and DPT-based prediction heads (heads/*.py).
• Streaming capabilities: GCTStream enables real-time causal inference with memory-efficient KV-caching, while GCTStreamWindow handles arbitrarily long sequences through overlapping window processing.
• Configuration: Default parameters use 518×518 images, 14×14 patches, 1024 embedding dimensions, and DINOv2 ViT-L14 registration tokens.
• Temporal consistency: Optional 3-D Rotary Positional Encoding (RoPE) maintains geometric coherence across video frames.

Frequently Asked Questions

What is the Geometric Context Transformer used for?

The Geometric Context Transformer powers 3D scene understanding in robotics and AR/VR applications. It processes monocular video to simultaneously estimate camera trajectories (9-DOF poses), dense depth maps, and world-coordinate point clouds, making it suitable for real-time SLAM, mapping, and spatial computing pipelines.

How does GCT handle long video sequences?

The architecture provides two mechanisms for long sequences. First, GCTStream uses a KV-cache with configurable sliding windows to limit memory growth linearly rather than quadratically with sequence length. Second, GCTStreamWindow automatically segments videos into overlapping windows, processes each independently, and aligns them using learned similarity transformations before stitching the results.

What is the role of 3-D RoPE in the architecture?

3-D Rotary Positional Encoding (enabled via enable_3d_rope=True) injects temporal sinusoidal embeddings into the attention computation. This allows the transformer to maintain geometric consistency across time by encoding relative temporal positions directly into the attention scores, improving trajectory smoothness and reducing drift in long sequences.

Which prediction heads are included in the GCT architecture?

The architecture includes three specialized heads: a depth head and point head (both DPT-style dense prediction heads in dpt_head.py outputting 2 and 4 channels respectively), and a camera head (CameraCausalHead in camera_head.py) that causally refines 9-dimensional pose encodings. Each head can be independently enabled or disabled via constructor flags (enable_depth, enable_point, enable_camera).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}