# Geometric Context Transformer in LingBot-Map: Architecture and Implementation Guide

> Explore LingBot-Map's Geometric Context Transformer architecture. Learn how this Vision-Transformer model generates camera poses, depth maps, and 3D points from video streams. Get the implementation guide.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: architecture
- Published: 2026-07-05

---

**The Geometric Context Transformer (GCT) is a Vision-Transformer-based architecture that processes video streams to output camera poses, dense depth maps, and 3D world points through a modular pipeline of patch embedding, transformer aggregation with optional KV-caching and 3D Rotary Positional Encoding (RoPE), and DPT-based prediction heads.**

LingBot-Map builds its 3D scene-understanding backbone on the Geometric Context Transformer. This architecture transforms sequential RGB frames into coherent geometric contexts—camera poses, dense depth, and point clouds—using a three-stage pipeline designed for both accuracy and streaming efficiency. The implementation leverages DINOv2-derived patch embeddings, multi-scale transformer blocks, and specialized dense prediction heads to deliver real-time 3D geometry estimation.

## Core Components of the Geometric Context Transformer

The Geometric Context Transformer architecture consists of three primary components orchestrated by the abstract `GCTBase` class. These modules handle tokenization, temporal feature aggregation, and geometric decoding.

### Patch Embedding and Tokenization

The input pipeline begins with a **patch embedding layer** that converts raw RGB images into transformer-compatible token sequences.

- **Implementation**: [`lingbot_map/layers/patch_embed.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/patch_embed.py)
- **Default configuration**: DINOv2 ViT-L/14 with registers (`dinov2_vitl14_reg`)
- **Key parameters**: `img_size=518`, `patch_size=14`, `embed_dim=1024`

This component slices each input frame into non-overlapping patches and projects them into a high-dimensional embedding space, producing a sequence of tokens that feed into the transformer aggregator.

### Transformer Aggregator with Multi-Scale Processing

The **aggregator** forms the computational core of the Geometric Context Transformer, processing token sequences through multi-scale transformer blocks.

- **Implementation**: [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py)
- **Key features**:
  - **3D Rotary Positional Encoding (RoPE)**: Injects temporal sinusoids into attention scores when `enable_3d_rope=True`, maintaining geometric consistency across frames
  - **KV-cache**: Enables causal attention for streaming inference, where each frame attends only to cached tokens from previous frames via `kv_cache_sliding_window` parameters
  - **FlashInfer integration**: Optional optimized attention kernels when `use_flashinfer=True`

The aggregator handles both bidirectional processing for initialization frames and causal processing for streaming video, dramatically reducing GPU memory requirements during online inference.

### Prediction Heads for Geometric Decoding

The aggregated tokens pass through task-specific **prediction heads** that decode features into concrete geometric outputs:

- **Depth head**: Located in [`lingbot_map/heads/dpt_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/dpt_head.py), this Dense Prediction Transformer (DPT) head outputs a 2-channel map containing depth values and confidence scores.
- **Point head**: Also implemented in [`dpt_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/dpt_head.py), but configured for 4-channel output (XYZ coordinates plus confidence), generating world-space point clouds.
- **Camera head**: Implemented in [`lingbot_map/heads/camera_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/camera_head.py) as a `CameraCausalHead`, this causal transformer refines 9-dimensional pose encodings (camera center plus quaternion) per frame using sliding window attention.

All heads share the same embedding dimension (`2 * embed_dim`) and operate on the patch-wise token stream produced by the aggregator.

## The GCTBase Abstraction

The `GCTBase` class in [`lingbot_map/models/gct_base.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_base.py) serves as the architectural scaffold, defining the common forward pipeline and hyper-parameter management.

The base class initializes with configuration flags for enabling specific outputs:

```python
class GCTBase(nn.Module, PyTorchModelHubMixin, ABC):
    def __init__(self,
                 img_size: int = 518,
                 patch_size: int = 14,
                 embed_dim: int = 1024,
                 patch_embed: str = 'dinov2_vitl14_reg',
                 enable_camera: bool = True,
                 enable_point: bool = True,
                 enable_depth: bool = True,
                 enable_3d_rope: bool = False,
                 use_gradient_checkpoint: bool = True):
        # Builds aggregator and heads based on flags

        self.aggregator = self._build_aggregator()
        self.camera_head = self._build_camera_head() if enable_camera else None
        self.point_head = self._build_point_head() if enable_point else None
        self.depth_head = self._build_depth_head() if enable_depth else None

```

The forward implementation (lines 87-120 in [`gct_base.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_base.py)) first calls `_aggregate_features` (implemented by concrete subclasses) and then executes the `_predict_*` helpers to generate the final prediction dictionary containing `pose_enc`, `depth`, `world_points`, and associated confidence maps.

## Streaming Variants for Real-Time Processing

LingBot-Map provides two concrete implementations of the Geometric Context Transformer optimized for different inference scenarios.

### GCTStream for Online Video Processing

`GCTStream` in [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py) inherits from `GCTBase` and specializes the aggregator for causal streaming:

```python
class GCTStream(GCTBase):
    def _build_aggregator(self) -> nn.Module:
        return AggregatorStream(
            img_size=self.img_size,
            patch_size=self.patch_size,
            embed_dim=self.embed_dim,
            patch_embed=self.patch_embed,
            use_flashinfer=not self.use_sdpa,
            kv_cache_sliding_window=self.kv_cache_sliding_window,
            # ... additional parameters

        )

```

The `inference_streaming` method (lines 459-511) implements the following logic:
1. Normalizes input frames (shape `(B, S, 3, H, W)`)
2. Cleans the KV cache for stateful processing
3. Processes initial scale frames with bidirectional attention
4. Streams subsequent frames with causal attention, optionally skipping KV-cache updates for non-keyframes based on `keyframe_interval`

### GCTStreamWindow for Long-Sequence Handling

`GCTStreamWindow` in [`lingbot_map/models/gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py) extends the streaming capability to arbitrarily long videos through windowed processing.

The `inference_windowed` method (lines 1120-1190) implements:
- **Windowing**: Splits long sequences into overlapping chunks (configured via `window_size` and `overlap_size`)
- **Alignment**: Uses `_pairwise_alignment` to compute learned similarity transforms between adjacent windows
- **Stitching**: Concatenates results via `_stitch_windows` to maintain a global coordinate frame

This approach allows the Geometric Context Transformer to process long videos while preserving global consistency, automatically managing KV cache resets between windows.

## Implementation Examples

### Instantiating a Streaming GCT Model

Configure the Geometric Context Transformer for temporal consistency and efficient inference:

```python
from lingbot_map.models.gct_stream import GCTStream

model = GCTStream(
    img_size=518,
    patch_size=14,
    embed_dim=1024,
    patch_embed='dinov2_vitl14_reg',
    enable_camera=True,
    enable_depth=True,
    enable_point=False,
    enable_3d_rope=True,          # Enable temporal consistency

    sliding_window_size=64,       # Attention window size

    kv_cache_sliding_window=64,    # KV cache limit

    kv_cache_scale_frames=8,
)

model = model.cuda().eval()

```

### Running Streaming Inference

Process video tensors with causal attention and keyframe subsampling:

```python
import torch

# Input: (Batch, Sequence, Channels, Height, Width)

frames = torch.randn(1, 120, 3, 518, 518).cuda()

preds = model.inference_streaming(
    images=frames,
    num_scale_frames=1,
    keyframe_interval=4,
    output_device=torch.device('cpu')
)

# Extract predictions

pose_enc = preds['pose_enc']           # (B, S, 9)

depth = preds['depth']                 # (B, S, H, W, 1)

world_points = preds['world_points']   # (B, S, H, W, 3)

```

### Processing Long Videos with Windowed Inference

Handle arbitrarily long sequences using the windowed variant:

```python
preds = model.inference_windowed(
    images=frames,
    window_size=16,               # 16 keyframes per window

    overlap_size=8,               # 8 frame overlap

    keyframe_interval=2,
    flow_threshold=1.0,           # Flow-based keyframe selection

    output_device=torch.device('cpu')
)

```

## Summary

The Geometric Context Transformer in LingBot-Map delivers accurate 3D scene understanding through a modular, efficient architecture:

- **Three-stage pipeline**: DINOv2 patch embedding, transformer aggregation with optional 3D RoPE, and DPT-based prediction heads
- **Streaming optimization**: KV-cache and causal attention in `GCTStream` enable real-time processing with constant memory usage
- **Long-sequence support**: `GCTStreamWindow` provides windowed inference with automatic alignment and stitching for arbitrary video lengths
- **Flexible outputs**: Configurable heads for camera pose (9-DOF), depth estimation, and world point cloud generation
- **Source files**: Core logic resides in [`lingbot_map/models/gct_base.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_base.py), [`gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream.py), and [`gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream_window.py), with components in [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py) and `lingbot_map/heads/`

## Frequently Asked Questions

### What is the difference between GCTStream and GCTStreamWindow?

**GCTStream** processes video sequentially using a KV-cache for causal attention, ideal for online streaming applications with fixed memory budgets. **GCTStreamWindow** extends this capability by splitting long videos into overlapping windows, aligning them via learned similarity transforms (`_pairwise_alignment`), and stitching results to maintain global geometric consistency across arbitrary sequence lengths.

### How does the 3D Rotary Positional Encoding (RoPE) improve temporal consistency?

When `enable_3d_rope=True`, the Geometric Context Transformer injects temporal sinusoidal encodings into the attention mechanism. This 3D RoPE modifies attention scores based on relative temporal positions, helping the model maintain consistent geometric relationships across frames without requiring explicit temporal convolutions or recurrent states.

### Why does the architecture use DINOv2 for patch embedding?

The patch embedding layer in [`lingbot_map/layers/patch_embed.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/patch_embed.py) leverages DINOv2 ViT-L/14 with registers by default because DINOv2 provides robust, pre-trained visual features that capture both low-level textures and high-level semantic information. This initialization significantly improves convergence and final accuracy for downstream geometric prediction tasks compared to training from scratch.

### What are the memory benefits of KV-caching in the Geometric Context Transformer?

The KV-cache implementation in `AggregatorStream` stores only the key and value tensors from previous frames rather than recomputing attention with the full history. During streaming inference (as implemented in `GCTStream`), this reduces memory complexity from quadratic in sequence length to constant (bounded by `kv_cache_sliding_window`), enabling the processing of infinite video streams on limited GPU memory.