# How LingBot-Map's Geometric Context Transformer Works: Architecture and Implementation

> Explore LingBot-Map's Geometric Context Transformer architecture to convert RGB video into 3D geometric context like camera poses, depth maps, and point clouds. Understand its three-stage pipeline and implementation.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: architecture
- Published: 2026-07-06

---

**The Geometric Context Transformer (GCT) is a Vision-Transformer-style network that converts RGB video streams into coherent 3D geometric context—including camera poses, dense depth maps, and world-coordinate point clouds—through a three-stage pipeline of patch embedding, transformer aggregation with optional KV-caching, and task-specific prediction heads.**

LingBot-Map builds its scene-understanding backbone on the **Geometric Context Transformer (GCT)** architecture. This system processes temporal sequences of images to produce frame-wise geometric predictions suitable for SLAM, mapping, and AR/VR pipelines. The implementation resides in the `Robbyant/lingbot-map` repository, where the core logic is split between abstract base classes, streaming variants, and specialized prediction heads.

## Core Architecture Components

The GCT architecture comprises three integrated components that transform raw pixels into structured 3D geometry.

### Patch Embedding and Tokenization

The input pipeline begins in [`lingbot_map/layers/patch_embed.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/patch_embed.py), where a **DINOv2-derived patch embedder** converts each input image into a sequence of tokens. By default, the system uses `patch_embed='dinov2_vitl14_reg'` with a configuration of `img_size=518`, `patch_size=14`, and `embed_dim=1024`. This tokenization step produces a sequence of patch embeddings that serve as the input to the transformer aggregator.

### Transformer Aggregation

The central processing engine lives in [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py). The `AggregatorStream` class implements multi-scale transformer blocks that process the token sequence with two key optimizations:

- **3-D Rotary Positional Encoding (RoPE)**: When `enable_3d_rope=True`, the aggregator injects temporal sinusoids into attention scores to maintain geometric consistency across frames.
- **KV-Cache for Streaming**: The `kv_cache_sliding_window` parameter enables causal attention during inference, allowing each new frame to attend only to cached tokens from previous frames rather than recomputing the full sequence attention.

The aggregator supports `use_flashinfer` for optimized memory access and integrates with PyTorch's SDPA (Scaled Dot-Product Attention) when configured.

### Prediction Heads

Located in `lingbot_map/heads/`, the output layers decode aggregated tokens into concrete geometric quantities:

- **Depth Head** ([`lingbot_map/heads/dpt_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/dpt_head.py)): A Dense Prediction Transformer (DPT) head that outputs a 2-channel map containing `depth` and `depth_confidence`.
- **Point Head** ([`lingbot_map/heads/dpt_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/dpt_head.py)): Another DPT variant outputting 4 channels (XYZ coordinates plus confidence) for world-coordinate point clouds.
- **Camera Head** ([`lingbot_map/heads/camera_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/camera_head.py)): A `CameraCausalHead` that implements causal transformer decoding to produce a 9-dimensional pose encoding (camera center plus quaternion) per frame.

All heads operate on an embedding dimension of `2 * embed_dim` and share the patch-wise token stream produced by the aggregator.

## The GCTBase Abstraction

The file [`lingbot_map/models/gct_base.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_base.py) defines the `GCTBase` abstract class that serves as the architectural scaffold for all GCT variants. This class stores hyper-parameters—including `img_size`, `patch_size`, and `embed_dim`—and constructs the aggregator and heads through factory methods:

```python
class GCTBase(nn.Module, PyTorchModelHubMixin, ABC):
    def __init__(self,
                 img_size: int = 518,
                 patch_size: int = 14,
                 embed_dim: int = 1024,
                 patch_embed: str = 'dinov2_vitl14_reg',
                 enable_camera: bool = True,
                 enable_point: bool = True,
                 enable_depth: bool = True,
                 enable_3d_rope: bool = False,
                 use_gradient_checkpoint: bool = True):
        ...
        self.aggregator = self._build_aggregator()
        self.camera_head = self._build_camera_head() if enable_camera else None
        self.point_head = self._build_point_head() if enable_point else None
        self.depth_head = self._build_depth_head() if enable_depth else None

```

The forward pipeline (lines 87-120 in [`gct_base.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_base.py)) first calls `_aggregate_features` (implemented by subclasses) and then executes the `_predict_*` helpers to return a dictionary containing `pose_enc`, `depth`, `depth_conf`, `world_points`, and `world_points_conf`.

## Streaming Inference Variants

For real-time applications, LingBot-Map provides two concrete implementations of `GCTBase` optimized for streaming video.

### GCTStream

The `GCTStream` class in [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py) specializes the base class for causal, frame-by-frame processing. It constructs an `AggregatorStream` with KV-cache support:

```python
class GCTStream(GCTBase):
    def _build_aggregator(self) -> nn.Module:
        return AggregatorStream(
            img_size=self.img_size,
            patch_size=self.patch_size,
            embed_dim=self.embed_dim,
            patch_embed=self.patch_embed,
            use_flashinfer=not self.use_sdpa,
            kv_cache_sliding_window=self.kv_cache_sliding_window,
            ...
        )

```

The `inference_streaming` method (lines 459-511) handles the end-to-end flow: it normalizes inputs, cleans the KV-cache, processes initial scale frames with bidirectional attention, then streams subsequent frames while optionally skipping KV-cache writes for non-keyframes based on the `keyframe_interval` parameter.

### GCTStreamWindow

For arbitrarily long sequences, `GCTStreamWindow` in [`lingbot_map/models/gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py) extends the streaming core with windowed inference. It splits video into overlapping windows, processes each with a fresh KV-cache, and aligns results via `_pairwise_alignment` (a learned similarity transform) before stitching them together with `_stitch_windows` (lines 1120-1190). This approach maintains a global coordinate frame while bounding memory usage.

## End-to-End Inference Examples

### Instantiating a Streaming Model

```python
from lingbot_map.models.gct_stream import GCTStream

model = GCTStream(
    img_size=518,
    patch_size=14,
    embed_dim=1024,
    patch_embed='dinov2_vitl14_reg',
    enable_camera=True,
    enable_depth=True,
    enable_point=False,
    enable_3d_rope=True,          # Enable temporal consistency

    sliding_window_size=64,       # Attention window size

    kv_cache_sliding_window=64,   # Cache for causal attention

    kv_cache_scale_frames=8,
)
model = model.cuda().eval()

```

This configuration builds the aggregator and heads exactly as defined in [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py) (lines 31-70).

### Running Streaming Inference

```python
import torch

# Input: (B, S, 3, H, W) with values in [0, 1]

frames = torch.randn(1, 120, 3, 518, 518).cuda()

preds = model.inference_streaming(
    images=frames,
    num_scale_frames=1,
    keyframe_interval=4,
    output_device=torch.device('cpu')
)

# Extract predictions

pose_enc = preds['pose_enc']         # (B, S, 9)

depth = preds['depth']               # (B, S, H, W, 1)

world_points = preds['world_points'] # (B, S, H, W, 3)

```

The method internally manages the KV-cache state and applies 3-D RoPE to maintain temporal geometric consistency.

### Processing Long Videos with Windowing

```python
preds = model.inference_windowed(
    images=frames,
    window_size=16,               # Keyframes per window

    overlap_size=8,               # Overlapping frames between windows

    keyframe_interval=2,
    flow_threshold=1.0,           # Enable optical-flow keyframe selection

    output_device=torch.device('cpu')
)

```

This routine automatically handles window alignment and stitching, producing globally consistent predictions across the entire sequence.

## Summary

- **GCT** processes RGB video through DINOv2 patch embedding, transformer aggregation with optional 3-D RoPE, and DPT-based prediction heads to output camera poses, depth, and 3D points.
- The **GCTBase** abstract class in [`lingbot_map/models/gct_base.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_base.py) defines the common forward pipeline and head construction, while concrete subclasses handle specific inference modes.
- **GCTStream** enables real-time causal inference with KV-caching and sliding-window attention, drastically reducing GPU memory for long sequences.
- **GCTStreamWindow** extends streaming with automatic windowing, alignment (`_pairwise_alignment`), and stitching (`_stitch_windows`) to handle arbitrarily long videos while maintaining global geometric consistency.
- Prediction heads in `lingbot_map/heads/` decode tokens into 9-D pose encodings (camera center + quaternion), 2-channel depth maps, and 4-channel world point clouds (XYZ + confidence).

## Frequently Asked Questions

### What is the primary purpose of the Geometric Context Transformer in LingBot-Map?

The Geometric Context Transformer serves as the core 3D scene-understanding engine that processes monocular video to simultaneously estimate camera ego-motion, per-pixel depth, and 3D world coordinates. According to the `Robbyant/lingbot-map` source code, it combines Vision-Transformer architectures with geometric inductive biases to produce outputs compatible with downstream SLAM and dense mapping systems.

### How does the KV-cache improve streaming performance in GCT?

The KV-cache mechanism, implemented in [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py) and utilized by `GCTStream`, stores key and value tensors from previous frames during causal attention. This allows the model to process each new frame by attending only to the cached history rather than recomputing attention over the entire sequence, reducing GPU memory from quadratic to linear complexity with respect to sequence length.

### What is the difference between GCTStream and GCTStreamWindow?

**GCTStream** processes video as a single continuous stream using a fixed-size sliding window and KV-cache, suitable for online robotics applications. **GCTStreamWindow**, defined in [`lingbot_map/models/gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py), automatically segments long videos into overlapping chunks, processes each chunk independently, and aligns them using `_pairwise_alignment` before stitching with `_stitch_windows`, making it ideal for offline batch processing of long recordings.

### Which geometric outputs does the Geometric Context Transformer produce?

The transformer outputs four primary geometric quantities through its heads: `pose_enc` (9-dimensional camera pose encoding from [`lingbot_map/heads/camera_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/camera_head.py)), `depth` and `depth_conf` (metric depth with confidence from the DPT head), and `world_points` with `world_points_conf` (3D coordinates in global frame from the point head). Utilities in [`lingbot_map/utils/pose_enc.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/pose_enc.py) and [`lingbot_map/utils/geometry.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/geometry.py) convert these encodings into standard camera matrices and point clouds.