# Geometric Context Transformer Architecture in LingBot-Map: A Technical Deep Dive

> Explore the Geometric Context Transformer architecture in LingBot-Map. This technical deep dive covers its Vision-Transformer-style network processing RGB video for camera pose, depth maps, and 3D point clouds.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: deep-dive
- Published: 2026-07-03

---

**The Geometric Context Transformer (GCT) is a Vision-Transformer-style network that processes RGB video streams to jointly estimate camera poses, dense depth maps, and 3D point clouds through patch embedding, streaming transformer aggregation with KV-caching, and specialized DPT-based prediction heads.**

The **Geometric Context Transformer** serves as the core 3D scene-understanding backbone in the LingBot-Map repository. This architecture combines a DINOv2-derived patch embedder with causal transformer blocks to generate coherent geometric context from video frames. Written in PyTorch, the implementation provides both real-time streaming and windowed inference modes for handling arbitrarily long sequences.

## Core Components of the Geometric Context Transformer

The GCT architecture consists of three interconnected stages orchestrated by the abstract `GCTBase` class. Each component handles a specific aspect of transforming raw pixels into structured 3D geometry.

### Patch Embedding and Tokenization

The input pipeline begins in [`lingbot_map/layers/patch_embed.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/patch_embed.py), where a DINOv2-derived patch embedder converts each RGB frame into a sequence of tokens. By default, the model uses `patch_embed='dinov2_vitl14_reg'` with an image size of 518 pixels and a patch size of 14, producing tokens with an embedding dimension of 1024.

This tokenization process allows the transformer to process visual information using self-attention mechanisms, similar to how Vision Transformers handle image classification tasks.

### Transformer Aggregator with KV-Cache

The central processing unit resides in [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py) within the `AggregatorStream` class. This component stacks multi-scale transformer blocks that process the token sequence temporally.

Key features of the aggregator include:

- **KV-Cache Mechanism**: Enables causal attention during streaming inference, where each new frame only attends to cached tokens from previous frames rather than recomputing the full sequence attention. This dramatically reduces GPU memory consumption during real-time processing.
- **3-D Rotary Positional Encoding (RoPE)**: When `enable_3d_rope=True`, the architecture injects temporal sinusoids into attention scores to maintain geometric consistency across time.
- **Sliding Window Management**: The `kv_cache_sliding_window` and `kv_cache_scale_frames` parameters control memory retention for long sequences.

### Prediction Heads for Geometry

The aggregated tokens flow into three specialized heads located in `lingbot_map/heads/`:

- **Depth Head** ([`dpt_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/dpt_head.py)): A Dense Prediction Transformer (DPT) head that outputs a 2-channel map containing depth values and confidence scores.
- **Point Head** ([`dpt_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/dpt_head.py)): Another DPT head with 4 output channels (XYZ coordinates plus confidence) for generating world-space point clouds.
- **Camera Head** ([`camera_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/camera_head.py)): A `CameraCausalHead` that performs iterative refinement on a 9-dimensional pose encoding (camera center plus quaternion) using causal transformer blocks.

All heads operate on a shared embedding dimension of `2 * embed_dim` and consume the patch-wise token stream produced by the aggregator.

## GCTBase: The Abstract Scaffold

The `GCTBase` class in [`lingbot_map/models/gct_base.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_base.py) defines the shared interface and initialization logic for all GCT variants. It handles hyper-parameter storage and head construction while leaving the aggregation implementation to subclasses.

```python
class GCTBase(nn.Module, PyTorchModelHubMixin, ABC):
    def __init__(self,
                 img_size: int = 518,
                 patch_size: int = 14,
                 embed_dim: int = 1024,
                 patch_embed: str = 'dinov2_vitl14_reg',
                 enable_camera: bool = True,
                 enable_point: bool = True,
                 enable_depth: bool = True,
                 enable_3d_rope: bool = False,
                 use_gradient_checkpoint: bool = True):
        super().__init__()
        self.aggregator = self._build_aggregator()
        self.camera_head = self._build_camera_head() if enable_camera else None
        self.point_head = self._build_point_head() if enable_point else None
        self.depth_head = self._build_depth_head() if enable_depth else None

```

The forward pass (lines 87-120 in [`gct_base.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_base.py)) first calls `_aggregate_features` (implemented by concrete subclasses) and then runs the four `_predict_*` helpers to generate the final prediction dictionary containing pose encodings, depth maps, and world points.

## Streaming Variants for Real-Time Inference

LingBot-Map provides two concrete implementations of `GCTBase` optimized for different inference scenarios.

### GCTStream for Causal Processing

`GCTStream` in [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py) specializes the base class for online video processing. It constructs an `AggregatorStream` with support for KV-caching and optional Flash Infer attention:

```python
class GCTStream(GCTBase):
    def _build_aggregator(self) -> nn.Module:
        return AggregatorStream(
            img_size=self.img_size,
            patch_size=self.patch_size,
            embed_dim=self.embed_dim,
            patch_embed=self.patch_embed,
            pretrained_path=self.pretrained_path,
            use_flashinfer=not self.use_sdpa,
            kv_cache_sliding_window=self.kv_cache_sliding_window,
            enable_3d_rope=self.enable_3d_rope
        )

```

The causal attention mechanism ensures that each frame only attends to previous tokens, making the model suitable for real-time SLAM and odometry applications.

### GCTStreamWindow for Long Sequences

`GCTStreamWindow` (located in [`lingbot_map/models/gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py)) extends the streaming capability to arbitrarily long videos by processing overlapping windows. It splits sequences into manageable chunks, aligns them via `_pairwise_alignment`, and stitches results using `_stitch_windows`. This approach maintains a global coordinate frame while circumventing memory limitations on long videos.

## End-to-End Inference Pipeline

### Instantiating a Streaming Model

To create a GCT model with temporal consistency enabled:

```python
from lingbot_map.models.gct_stream import GCTStream

model = GCTStream(
    img_size=518,
    patch_size=14,
    embed_dim=1024,
    patch_embed='dinov2_vitl14_reg',
    enable_camera=True,
    enable_depth=True,
    enable_point=False,
    enable_3d_rope=True,          # Enable temporal consistency

    sliding_window_size=64,        # KV cache window size

    kv_cache_sliding_window=64,
    kv_cache_scale_frames=8
).cuda().eval()

```

The constructor chains together the patch embedder, aggregator, and prediction heads exactly as defined in [`gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream.py) (lines 31-70).

### Running Streaming Inference

Process video sequences with causal KV-caching:

```python
import torch

# frames shape: (Batch, Sequence, Channels, Height, Width)

frames = torch.randn(1, 120, 3, 518, 518).cuda()

preds = model.inference_streaming(
    images=frames,
    num_scale_frames=1,
    keyframe_interval=4,
    output_device=torch.device('cpu')
)

# Extract predictions

pose_enc = preds['pose_enc']        # (B, S, 9) - camera centers and quaternions

depth = preds['depth']              # (B, S, H, W, 1)

world_points = preds['world_points'] # (B, S, H, W, 3)

```

The `inference_streaming` method (lines 459-511 in [`gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream.py)) handles input normalization, KV-cache initialization, and keyframe selection logic. It processes scale frames with bidirectional attention before switching to causal streaming mode.

### Processing Long Videos with Windowed Inference

For sequences exceeding GPU memory limits:

```python
preds = model.inference_windowed(
    images=frames,
    window_size=16,               # 16 keyframes per window

    overlap_size=8,               # 8 frames of overlap

    keyframe_interval=2,
    flow_threshold=1.0,           # Enable flow-based keyframe selection

    output_device=torch.device('cpu')
)

```

The windowed routine (lines 1120-1190 in [`gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream_window.py)) automatically manages window boundaries, runs each segment with a fresh KV cache, and performs global alignment across windows.

## Summary

- **Geometric Context Transformer** combines Vision-Transformer architecture with 3D geometric prediction heads for simultaneous camera pose estimation, depth prediction, and point cloud generation.
- **Three-stage pipeline**: Patch embedding ([`patch_embed.py`](https://github.com/Robbyant/lingbot-map/blob/main/patch_embed.py)), transformer aggregation with KV-cache ([`stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/stream.py)), and DPT-based prediction heads (`heads/*.py`).
- **Streaming capabilities**: `GCTStream` enables real-time causal inference with memory-efficient KV-caching, while `GCTStreamWindow` handles arbitrarily long sequences through overlapping window processing.
- **Configuration**: Default parameters use 518×518 images, 14×14 patches, 1024 embedding dimensions, and DINOv2 ViT-L14 registration tokens.
- **Temporal consistency**: Optional 3-D Rotary Positional Encoding (RoPE) maintains geometric coherence across video frames.

## Frequently Asked Questions

### What is the Geometric Context Transformer used for?

The Geometric Context Transformer powers 3D scene understanding in robotics and AR/VR applications. It processes monocular video to simultaneously estimate camera trajectories (9-DOF poses), dense depth maps, and world-coordinate point clouds, making it suitable for real-time SLAM, mapping, and spatial computing pipelines.

### How does GCT handle long video sequences?

The architecture provides two mechanisms for long sequences. First, `GCTStream` uses a KV-cache with configurable sliding windows to limit memory growth linearly rather than quadratically with sequence length. Second, `GCTStreamWindow` automatically segments videos into overlapping windows, processes each independently, and aligns them using learned similarity transformations before stitching the results.

### What is the role of 3-D RoPE in the architecture?

**3-D Rotary Positional Encoding** (enabled via `enable_3d_rope=True`) injects temporal sinusoidal embeddings into the attention computation. This allows the transformer to maintain geometric consistency across time by encoding relative temporal positions directly into the attention scores, improving trajectory smoothness and reducing drift in long sequences.

### Which prediction heads are included in the GCT architecture?

The architecture includes three specialized heads: a **depth head** and **point head** (both DPT-style dense prediction heads in [`dpt_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/dpt_head.py) outputting 2 and 4 channels respectively), and a **camera head** (`CameraCausalHead` in [`camera_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/camera_head.py)) that causally refines 9-dimensional pose encodings. Each head can be independently enabled or disabled via constructor flags (`enable_depth`, `enable_point`, `enable_camera`).