# Understanding the Data Model for LingBot-Map: Inside the GCT Architecture

> Explore the LingBot-Map data model. Discover how GCT processes RGB images to output camera pose, depth maps, and 3D point clouds with a unified 9-dimensional pose encoding.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: data-model-explanation
- Published: 2026-07-19

---

**The LingBot-Map repository implements a unified GCT (Generalized Camera Transformer) data model centered on the `GCTBase` abstract class, which processes RGB images through token aggregation to simultaneously output camera pose, depth maps, and 3D point clouds using a compact 9-dimensional pose encoding.**

The LingBot-Map project by Robbyant provides a vision-centric mapping system built around a flexible transformer-based data model. At its foundation lies the **Generalized Camera Transformer (GCT)** architecture, which unifies multiple geometric prediction tasks within a single tensor representation. This article examines the core data structures, processing pipelines, and implementation details that define the data model for LingBot-Map according to the source code.

## Core Architecture of the LingBot-Map Data Model

### The GCTBase Abstract Class

The data model for LingBot-Map is anchored by the abstract base class **`GCTBase`**, defined in [`lingbot_map/models/gct_base.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_base.py). This class establishes the common data flow and component structure shared across all model variants. It orchestrates four primary subsystems: input normalization, token aggregation, pose encoding, and multi-task prediction heads.

The base class provides concrete implementations for `_normalize_input`, which handles both single-frame and batched image tensors, and abstract methods like `_build_aggregator` that subclasses must implement. This design pattern ensures consistent data handling while allowing flexibility in the underlying transformer architecture.

### Token Aggregation Pipeline

Central to the data model is the **aggregator** component, responsible for converting image patches into feature tokens. The standard aggregator processes complete sequences, while the streaming variant—implemented in [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py) as **`AggregatorStream`**—maintains a KV-cache for online inference. This aggregation step transforms visual input into a token sequence where spatial information is encoded as transformer-compatible embeddings.

## Data Flow and Processing Pipeline

### From Images to Token Sequences

The data flow begins with **`_normalize_input`** processing raw RGB tensors. Images undergo patch embedding to generate token sequences that preserve spatial relationships. The model handles variable batch sizes and optional query points, normalizing inputs before they reach the transformer backbone.

This tokenization process creates a unified tensor representation that serves as the intermediate data structure for all downstream predictions. The sequence length depends on the image resolution and patch size, with each token representing a localized region of the visual input.

### The 9-Dimensional Pose Encoding Scheme

LingBot-Map employs a compact **9-dimensional pose encoding** format: **`[T_x, T_y, T_z, q_w, q_x, q_y, q_z, fov_h, fov_w]`**. This vector concatenates 3D translation coordinates, quaternion rotation components, and horizontal/vertical field-of-view parameters.

The conversion utilities in [`lingbot_map/utils/pose_enc.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/pose_enc.py) provide bidirectional mapping between this encoding and standard camera matrices. The functions **`extri_intri_to_pose_encoding`** and **`pose_encoding_to_extri_intri`** handle the mathematical transformations, requiring image dimensions to properly scale the field-of-view parameters.

### Multi-Task Prediction Heads

The aggregated tokens feed into specialized prediction heads constructed via **`_build_camera_head`**, **`_build_depth_head`**, **`_build_point_head`**, and **`_build_local_point_head`**. Critically, the data model reserves the **first token** of the sequence for camera pose prediction, while subsequent tokens handle depth and 3D point estimation.

The depth and point heads—implemented in [`lingbot_map/heads/dpt_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/dpt_head.py) using the DPT (Dense Prediction Transformer) architecture—output **2-channel depth maps** and **4-channel point clouds** respectively. The camera head in [`lingbot_map/heads/camera_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/camera_head.py) provides iterative refinement capabilities for pose estimation.

## Streaming Implementation and KV-Cache Support

For online mapping applications, **`GCTStream`** in [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py) extends the base data model with streaming capabilities. This variant utilizes **`AggregatorStream`** to maintain key-value caches across sequential frames, enabling constant-time inference regardless of sequence length. The streaming data model preserves the same tensor structures as the batch variant while adding temporal state management for real-time applications.

## Practical Implementation Examples

### Instantiating a Model and Running Inference

```python
import torch
from lingbot_map.models.gct_stream import GCTStream

# Initialise a streaming GCT model (default config)

model = GCTStream()

# Dummy batch of 2 RGB images (B, C, H, W)

images = torch.randn(2, 3, 518, 518)

# Forward pass returns a dict with camera pose, depth and point predictions

outputs = model(images)

# Access the camera pose encoding (post‑activation)

pose_enc = outputs["pose_enc"]          # shape [B, 9]

# Convert to extrinsics / intrinsics (needs image size)

extrinsics, intrinsics = model.pose_encoding_to_extri_intri(
    pose_enc, image_size_hw=(518, 518)
)

```

### Converting Between Pose Encodings and Camera Matrices

```python
from lingbot_map.utils.pose_enc import extri_intri_to_pose_encoding, pose_encoding_to_extri_intri
import torch

# Example extrinsic (B, S, 3, 4) and intrinsic (B, S, 3, 3) tensors

extr = torch.randn(1, 1, 3, 4)
intr = torch.randn(1, 1, 3, 3)

# Encode

pose_enc = extri_intri_to_pose_encoding(extr, intr, image_size_hw=(256, 512))

# Decode back (requires image size)

extr_dec, intr_dec = pose_encoding_to_extri_intri(pose_enc, image_size_hw=(256, 512))

```

## Summary

- The data model for LingBot-Map centers on the **`GCTBase`** class in [`lingbot_map/models/gct_base.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_base.py), providing an abstract foundation for transformer-based geometric prediction that unifies camera pose, depth, and point cloud estimation.
- A compact **9-dimensional pose encoding** (`[T_x,T_y,T_z, q_w,q_x,q_y,q_z, fov_h, fov_w]`) stores translation, quaternion rotation, and field-of-view parameters, with conversion utilities located in [`lingbot_map/utils/pose_enc.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/pose_enc.py).
- The architecture processes **image patches into token sequences** through configurable aggregators, reserving the first token for camera pose predictions while subsequent tokens handle depth and 3D point generation.
- **DPT-based prediction heads** in [`lingbot_map/heads/dpt_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/dpt_head.py) generate 2-channel depth maps and 4-channel point clouds, constructed via methods like `_build_depth_head` and `_build_point_head`.
- Streaming variants support online inference through KV-cache mechanisms implemented in [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py), enabling efficient temporal processing without increasing computational complexity.

## Frequently Asked Questions

### What is the primary data structure used in LingBot-Map's GCT model?

The primary data structure is the **token sequence** generated from image patch embeddings, processed through the `GCTBase` architecture. This sequence serves as the unified intermediate representation where the first token specifically encodes camera pose via the 9-dimensional format, while remaining tokens carry features for depth and point cloud prediction heads.

### How does LingBot-Map represent camera pose internally?

Camera pose is represented as a **9-dimensional tensor** `[T_x,T_y,T_z, q_w,q_x,q_y,q_z, fov_h, fov_w]` combining 3D translation coordinates, quaternion rotation components, and horizontal/vertical field-of-view values. The [`lingbot_map/utils/pose_enc.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/pose_enc.py) module provides `extri_intri_to_pose_encoding` and `pose_encoding_to_extri_intri` functions to convert between this compact encoding and standard 3x4 extrinsic and 3x3 intrinsic camera matrices.

### What distinguishes GCTBase from GCTStream in the LingBot-Map codebase?

**`GCTBase`** provides the standard transformer architecture designed for batch processing of independent frames, while **`GCTStream`**—found in [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py)—extends this with KV-cache support for streaming inference. The streaming variant utilizes `AggregatorStream` to maintain state across sequential frames, making it suitable for online SLAM and real-time mapping applications.

### Where are the depth and point prediction heads implemented?

These heads are implemented in **[`lingbot_map/heads/dpt_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/dpt_head.py)** using the Dense Prediction Transformer (DPT) architecture. The depth head outputs 2-channel tensors representing per-pixel depth values, while the point head generates 4-channel tensors encoding 3D coordinates. Both are instantiated through the base class methods `_build_depth_head` and `_build_point_head`, with the camera head separately implemented in [`lingbot_map/heads/camera_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/camera_head.py).