# How LingBot-Map Handles Coordinate Grounding with the Pose-Reference Window

> Discover how LingBot-Map grounds coordinates using a pose-reference window for robust global system alignment. Learn about its efficient keyframe and depth ratio techniques.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: how-to-guide
- Published: 2026-06-29

---

**LingBot-Map anchors each processing window to a pose-reference window containing jointly optimized scale frames, then aligns consecutive windows via similarity transforms computed from overlapping keyframes and depth ratios to maintain a single global coordinate system.**

Robbyant/lingbot-map processes arbitrarily long video sequences by splitting them into manageable windows that are processed independently. The critical challenge is maintaining a coherent world coordinate system across these windows, which the repository solves through a sophisticated coordinate grounding mechanism centered on the pose-reference window.

## The Windowed Processing Architecture

Long video sequences exceed GPU memory constraints when processed monolithically. LingBot-Map addresses this by dividing videos into independent windows that are processed sequentially. Without proper alignment, each window would define its own local coordinate system, causing drift and fragmentation in the global trajectory.

The architecture preserves global consistency by grounding every window to a **pose-reference window**—a set of initial scale frames that serve as a geometric anchor. This grounding happens through overlap-based alignment that computes a similarity transform (scale, rotation, and translation) mapping each window into the previous window's coordinate frame.

## Scale-Frame Reference Grounding

Each window begins with a predetermined number of **scale frames** defined by the `num_frame_for_scale` parameter (typically 4). These frames are processed together with jointly optimized camera poses according to the source code in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py).

This joint optimization ensures the pose-reference window provides a reliable geometric anchor for the remainder of the window. All subsequent frames in the window are predicted relative to this grounded reference, preventing the accumulation of local drift that would otherwise occur in independent window processing.

## Overlap-Based Alignment Between Windows

Consecutive windows share an **overlap region** containing at least one keyframe from each window. The alignment process extracts pose encodings (`pose_enc`) from these overlapping keyframes and converts them back to camera extrinsics and intrinsics using `pose_encoding_to_extri_intri`.

The system then computes a similarity transform consisting of:

- **Scale factor** (`s`) derived from depth predictions
- **Rotation matrix** (`R`) extracted from pose encodings
- **Translation vector** (`t`) bridging the coordinate systems

This transform is implemented in lines [1045-1082](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py#L1045-L1082) of [`gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream_window_v2.py), where the pairwise alignment routine extracts rotation and translation from pose encodings.

## Depth-Ratio Scale Estimation

Scale ambiguity presents a fundamental challenge in monocular reconstruction. LingBot-Map resolves this through the `_depth_ratio_scale` function, which estimates the scale factor from ratios of depth predictions in overlapping keyframes.

This depth-ratio approach makes the alignment robust to unknown scene scales, allowing the system to handle diverse environments without prior calibration. The scale factor aligns the metric scale of the current window with the previous window, ensuring metric consistency across the entire sequence.

## Warping Predictions and Global Stitching

Once the similarity transform is computed, the `_warp_predictions` function applies it to all window predictions including:

- Pose encodings
- Depth maps
- 3-D point clouds

The warped predictions are then concatenated using `_stitch_windows` (lines [1016-1044](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py#L1016-L1044)), yielding a globally consistent trajectory. This warping ensures that every point cloud and camera pose exists in a single, unified world coordinate system regardless of which window generated it.

## Implementation Details in gct_stream_window_v2.py

The core grounding logic resides in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py), specifically:

- **Lines 1045-1082**: Pairwise alignment routine that extracts rotation and translation from pose encodings and derives scale from depth ratios
- **Lines 1016-1044**: Similarity transform construction and application to window predictions
- **Lines 1028-1069**: Windowed inference orchestration handling both fixed-interval and flow-based keyframe selection

Supporting components include:

- [`lingbot_map/utils/pose_enc.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/pose_enc.py): Converts between camera extrinsics/intrinsics and the compact 9-dimensional pose encoding
- [`lingbot_map/utils/geometry.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/geometry.py): Provides `closed_form_inverse_se3` and other geometric helpers for pose transformations
- [`lingbot_map/heads/camera_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/camera_head.py): Supplies the causal camera head producing pose encodings consumed during grounding
- [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py): Manages the KV-cache and sliding-window attention for per-window processing

## Practical Code Example

The following example demonstrates windowed inference with pose-reference grounding:

```python
import torch
from lingbot_map.models.gct_stream_window_v2 import GCTStream

# Initialise the streaming model (defaults enable camera head, 3-D RoPE disabled)

model = GCTStream(
    img_size=518,
    patch_size=14,
    embed_dim=1024,
    enable_camera=True,
    enable_3d_rope=False,
    sliding_window_size=64,      # KV-cache window

    num_frame_for_scale=4,       # → pose-reference (scale) frames per window

)

# Dummy video tensor: [B, S, 3, H, W] in range [0, 1]

video = torch.rand(1, 120, 3, 518, 518)

# Windowed inference with a keyframe every 5 frames and flow-based keyframe detection

preds = model.inference_windowed(
    images=video,
    window_size=16,              # 16 keyframes per window (includes the 4 scale frames)

    overlap_keyframes=2,         # 2 keyframes overlap → robust alignment

    keyframe_interval=5,         # every 5th frame is stored in KV cache

    flow_threshold=2.0,          # optional flow-based keyframe gating

    max_non_keyframe_gap=30,     # force a keyframe after 30 non-keyframes

)

# Access globally aligned world points

world_points = preds["world_points"]    # shape: [B, S, H, W, 3]

pose_enc      = preds["pose_enc"]       # camera pose encodings for every frame

```

## Summary

LingBot-Map maintains global coordinate consistency across long video sequences through:

- **Pose-reference windows** that provide jointly optimized scale frames as geometric anchors for each processing window
- **Overlap-based alignment** that computes similarity transforms (scale, rotation, translation) between consecutive windows using shared keyframes
- **Depth-ratio scaling** that resolves metric scale ambiguity by comparing depth predictions in overlapping regions
- **Warping and stitching** operations that transform all predictions into a unified world coordinate system before concatenation

## Frequently Asked Questions

### What is the pose-reference window in LingBot-Map?

The pose-reference window consists of the first `num_frame_for_scale` frames (typically 4) in each processing window. These scale frames are processed together with jointly optimized camera poses to establish a reliable geometric anchor. According to the source code in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py), this reference window grounds all subsequent predictions in the window to a consistent local coordinate system before global alignment occurs.

### How does LingBot-Map handle scale ambiguity between windows?

LingBot-Map resolves scale ambiguity through the `_depth_ratio_scale` function, which estimates the scale factor from the ratio of depth predictions in overlapping keyframes. This approach makes the cross-window alignment robust to unknown scene scales, allowing the similarity transform to align the current window's metric scale with the previous window without requiring prior calibration or known baseline distances.

### Which source files manage the pose encoding conversions?

The conversion between camera extrinsics/intrinsics and the compact 9-dimensional pose encoding used during alignment is handled by [`lingbot_map/utils/pose_enc.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/pose_enc.py). Geometric transformations required for the similarity transforms, including `closed_form_inverse_se3`, are implemented in [`lingbot_map/utils/geometry.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/geometry.py), while the camera head producing these encodings resides in [`lingbot_map/heads/camera_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/camera_head.py).

### How many frames typically overlap between consecutive windows?

The overlap is configurable via the `overlap_keyframes` parameter, with the code example demonstrating 2 overlapping keyframes. These shared keyframes must contain at least one frame from each window to compute the similarity transform. The system requires sufficient overlap to extract reliable `pose_enc` values and depth ratios for computing the alignment transform that maps the current window into the previous window's coordinate frame.