# How LingBot-Map Uses the Anchor Context Mechanism for Coordinate Grounding

> Discover the anchor context mechanism for coordinate grounding in LingBot-Map. Learn how it selects keyframes, estimates scale, and computes transforms for accurate mapping.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: deep-dive
- Published: 2026-07-04

---

**The anchor context mechanism grounds coordinates by selecting mutual keyframes in overlapping temporal regions, estimating scale from depth ratios, and computing a similarity transform that maps each window into a common coordinate frame.**

The LingBot-Map system, available in the Robbyant/lingbot-map repository, processes continuous video streams by splitting them into overlapping temporal windows. To maintain consistent world-space reconstruction across these chunks, the implementation relies on an anchor context mechanism for coordinate grounding that aligns predictions using shared visual references and robust depth-based scale estimation.

## How the Anchor Context Mechanism Works

According to the source code in [`lingbot_map/models/gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py), the mechanism operates through six distinct stages when processing consecutive windows.

### Overlap Region Detection

The system first identifies the shared temporal region between the previous and current windows. As implemented in lines 786 and 788:

```python
start = max(total_prev - overlap, 0)
eff_overlap = min(overlap, total_prev-start, total_curr)

```

This calculates the effective overlap size (`eff_overlap`) and the starting index (`start`) for alignment.

### Keyframe Pairing in the Overlap

Both windows store a binary `is_keyframe` flag per frame. The code identifies frames designated as keyframes in both windows within the overlap region (lines 894-896):

```python
kf_prev_tail = is_kf_prev[0, start:start + eff_overlap].to(torch.bool)
kf_curr_head = is_kf_curr[0, :eff_overlap].to(torch.bool)
paired_mask = kf_prev_tail & kf_curr_head

```

The indices of these paired keyframes are extracted using (lines 998-999):

```python
paired_offsets = torch.nonzero(paired_mask, as_tuple=False).flatten()

```

### Anchor Selection Logic

The system selects the temporal anchor using a priority-based approach:

**If paired keyframes exist**, the last (most recent) common keyframe serves as the anchor (lines 1000-1002):

```python
anchor_offset = int(paired_offsets[-1].item())
idx_a = start + anchor_offset
idx_b = anchor_offset

```

**If no common keyframes exist**, the mechanism falls back to the first frame of the overlap (lines 1007-1008):

```python
idx_a = start
idx_b = 0

```

### Depth-Based Scale Estimation

The anchor frames provide depth samples from both windows. The `_depth_ratio_scale` method aggregates depth values across all paired keyframes to compute a robust median scale factor (lines 1027-1031):

```python
s_ab = GCTStream._depth_ratio_scale(
    da[:, prev_depth_idx, ..., 0],
    db[:, curr_depth_idx, ..., 0],
    batch_size, device,
)

```

This `s_ab` tensor represents the scale ratio between the two windows' coordinate systems.

### Similarity Transform Computation

Using the anchor poses (`Ra`, `ca`, `Rb`, `cb`) and the scale `s_ab`, the method constructs a full similarity transform (scale **s**, rotation **R**, translation **t**) that maps the current window into the previous window's coordinate system (lines 1018 and 1034):

```python
R_ab = torch.bmm(Ra, Rb.transpose(1, 2))
t_ab = ca - s_ab.unsqueeze(-1) * torch.bmm(R_ab, cb.unsqueeze(-1)).squeeze(-1)

```

### Application to Window Predictions

Finally, the `_warp_predictions` method applies the computed similarity transform to every prediction tensor of the current window, thereby grounding it to the anchor context of the previous window and ensuring consistent world-space geometry.

## Implementation in gct_stream_window.py

The core logic resides in [`lingbot_map/models/gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py) within the `GCTStream` class. The `_pairwise_alignment` method orchestrates the overlap detection, keyframe matching, and transform computation, while `_depth_ratio_scale` handles the robust scale estimation using median depth ratios across the batch.

## Practical Code Examples

### Manual Alignment Invocation

You can manually invoke the anchor alignment between two window predictions:

```python
import torch
from lingbot_map.models.gct_stream_window import GCTStream

# Dummy predictions for two overlapping windows (batch = 1)

prev_pred = {
    "pose_enc": torch.randn(1, 10, 7),     # [batch, frames, (x,y,z, qw,qx,qy,qz)]

    "is_keyframe": torch.randint(0, 2, (1, 10)),
    "depth": torch.randn(1, 10, 64, 64, 1)
}
curr_pred = {
    "pose_enc": torch.randn(1, 10, 7),
    "is_keyframe": torch.randint(0, 2, (1, 10)),
    "depth": torch.randn(1, 10, 64, 64, 1)
}

# Align current window to previous window (overlap = 4 frames)

scale, R, t = GCTStream._pairwise_alignment(
    prev_pred, curr_pred,
    overlap=4,
    batch_size=1,
    device=torch.device("cpu"),
    dtype=torch.float32,
)

# Apply the transform to the current predictions

aligned_curr = GCTStream._warp_predictions(curr_pred, R, t, scale, batch_size=1)

```

### High-Level Streaming API

In production pipelines, you typically use the high-level API that handles windowing automatically:

```python
stream = GCTStream(...)
stream.run(video)  # Internally splits into windows, applies anchor context alignment, and stitches

```

## Key Files and Architecture

- **[`lingbot_map/models/gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py)**: Contains the core `_pairwise_alignment` and `_depth_ratio_scale` implementations
- **[`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py)**: Optimized variant with minor performance improvements
- **[`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py)**: High-level orchestration class managing window creation and stitching
- **[`lingbot_map/utils/pose_enc.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/pose_enc.py)**: Quaternion-to-rotation matrix conversions for pose handling
- **[`benchmark/benchmark/core/loader.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/core/loader.py)**: Video data loading and feeding infrastructure

## Summary

- The anchor context mechanism processes overlapping temporal windows to maintain consistent world coordinates in LingBot-Map
- It selects anchors by finding mutual keyframes in the overlap region, falling back to the first overlap frame if necessary
- Depth ratios from paired keyframes provide robust scale estimation via median aggregation
- A similarity transform (scale, rotation, translation) aligns the current window to the previous window's coordinate frame
- The implementation in [`gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream_window.py) handles all stages from overlap detection to prediction warping

## Frequently Asked Questions

### What happens when no common keyframes exist in the overlap region?

If the boolean AND of the keyframe masks yields no valid pairs, the system defaults to the first frame of the overlap region as the anchor. This legacy fallback behavior ensures the system can still compute an alignment transform even when keyframe sampling is sparse, though it may be less robust than using a mutually detected keyframe.

### How does the mechanism handle scale ambiguity between windows?

The `_depth_ratio_scale` function aggregates depth values across all paired keyframes in the overlap and computes the median ratio between corresponding depth maps. This median-based approach provides robustness against outliers and transient occlusion artifacts, producing a reliable scale factor `s_ab` that accounts for the metric scale difference between the two windows' local coordinate systems.

### Why use keyframes instead of every frame for anchoring?

The binary `is_keyframe` flag filters the temporal stream to select only high-quality frames suitable for reliable pose estimation. By restricting anchor selection to frames designated as keyframes in both windows, the system avoids aligning on transitional or motion-blurred frames, resulting in more stable similarity transforms and reduced drift accumulation across the video sequence.

### What is the computational overhead of the anchor context alignment?

The alignment operates on the overlap region only (typically a small subset of the window), and the dominant cost is the batched matrix multiplication for rotation composition and the median calculation for scale estimation. As implemented in the `GCTStream` class, these operations utilize PyTorch's optimized CUDA kernels, making the anchor context mechanism computationally efficient relative to the per-window depth and pose estimation steps.