# How the Pose-Reference Window Enables Drift Correction in LingBot-Map

> LingBot-Map uses a pose-reference window to correct drift. Learn how it compares keyframes and warps predictions for a consistent coordinate system.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: how-to-guide
- Published: 2026-07-04

---

**The pose-reference window in LingBot-Map corrects accumulated drift by splitting video streams into overlapping keyframe windows, comparing pose encodings between shared anchor frames, and warping predictions using a learned similarity transform to maintain a globally consistent coordinate system.**

Robbyant/lingbot-map processes video streams in an online fashion, where sequential frame-by-frame estimation causes tiny pose errors to compound into severe drift over long sequences. The pose-reference window architecture solves this by periodically re-anchoring the scene using overlapping keyframes that bridge consecutive windows, allowing the model to align and stitch local trajectories into a single coherent global path.

## The Drift Problem in Online Video Processing

When a model processes thousands of frames sequentially, each new pose prediction builds upon the previous estimate. This recursive dependency means that even sub-pixel errors accumulate, causing the camera trajectory to drift away from its true path. Traditional solutions require storing all frames in memory for global optimization, which violates streaming constraints. LingBot-Map solves this through a **three-part architectural design** that balances memory efficiency with global consistency.

## Three-Pillar Architecture for Global Consistency

The repository implements drift correction through three coordinated components:

- **Anchor context** – Supplies the first `num_scale_frames` frames processed with bidirectional attention, establishing a solid global reference for scale and orientation.
- **Pose-reference window** – Splits the remaining sequence into overlapping windows of keyframes, using stored pose encodings to align and warp predictions between windows.
- **Trajectory memory** – Maintains a running record of aligned poses, enabling downstream modules like the point-cloud generator to operate on a globally consistent trajectory.

The **pose-reference window** serves as the active drift-correction mechanism, functioning like a sliding reset that stitches local window predictions into a unified global frame.

## How the Pose-Reference Window Corrects Drift

### Window Creation and Overlap Strategy

The windowing logic is implemented in `inference_windowed` (line 1025 in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py)). This method defines the streaming parameters that govern how the sequence is partitioned:

- `window_size` – Number of keyframes processed per window.
- `overlap_keyframes` – Shared frames between consecutive windows that serve as alignment anchors.
- `keyframe_interval` – Frequency of frames stored in the KV cache.

By maintaining overlapping keyframes between windows, the system creates common reference points that exist in both the previous and current coordinate frames.

### Alignment via Pose Encoding Comparison

When a new window finishes processing, the model executes `_pairwise_alignment` (line 1029) to estimate the similarity transform between the previous and current windows. This function extracts the **pose encodings** (`pose_enc`) from the last overlapping keyframe of the previous window and the first overlapping keyframe of the current window.

The alignment computes a similarity transform comprising:
- Scale **s**
- Rotation **R**
- Translation **t**

These parameters quantify exactly how much the current window has drifted relative to the global coordinate system established by previous windows.

### Warping and Stitching

After computing the alignment, `_warp_predictions` (line 1039) applies the similarity transform to all predictions in the current window—including pose, depth, and world points—effectively warping them into the coordinate frame of the first window.

The `_align_and_stitch_windows` method (line 1055) iterates over all windows, applies the pairwise alignment, and concatenates the results via `_stitch_windows`. This process removes accumulated drift between windows while preserving the local accuracy of windowed inference.

## Implementation in Code

You can enable drift correction using the command-line interface or the Python API.

### Command-Line Usage

Run the interactive demo in windowed mode to activate the pose-reference window:

```bash
python demo.py \
    --model_path /path/to/lingbot-map-long.pt \
    --image_folder example/university \
    --mode windowed \
    --window_size 128 \
    --overlap_keyframes 16 \
    --keyframe_interval 2 \
    --mask_sky

```

The `--overlap_keyframes` parameter ensures shared anchors exist between windows, which is essential for the drift correction mechanism to function.

### Python API Usage

For programmatic access, use the `GCTStream` class with `inference_windowed`:

```python
import torch
from lingbot_map.models.gct_stream_window_v2 import GCTStream

model = GCTStream().to('cuda')
images = torch.rand(1, 2000, 3, 518, 518).cuda()   # 2000-frame demo video

# Windowed inference with automatic drift correction

pred = model.inference_windowed(
    images,
    window_size=128,
    overlap_keyframes=16,
    keyframe_interval=2,
    output_device='cpu',
)

# Pose encodings are already drift-corrected

pose_enc = pred['pose_enc']   # shape [B, S, 9]

```

### Inspecting Alignment Between Windows

To verify the drift correction magnitude between consecutive windows:

```python
window_a = pred_a   # output dict from first window

window_b = pred_b   # output dict from second window

# Compute the similarity transform used for stitching

s, R, t = model._pairwise_alignment(
    window_a, 
    window_b,
    overlap=model._last_overlap_size,
    batch_size=window_a['pose_enc'].shape[0],
    device=window_a['pose_enc'].device,
    dtype=window_a['pose_enc'].dtype
)

print('Scale:', s.mean().item())
print('Rotation norm:', R.norm(p=2, dim=[1,2]).mean().item())
print('Translation norm:', t.norm(p=2, dim=1).mean().item())

```

## Summary

- **The pose-reference window** combats drift by re-anchoring the scene every `window_size` frames using overlapping keyframes as shared reference points.
- **Alignment** occurs via `_pairwise_alignment`, which compares 9-dimensional pose encodings (`pose_enc`) between overlapping frames to estimate scale, rotation, and translation differences.
- **Warping** via `_warp_predictions` applies the similarity transform to all predictions, stitching windows into a globally consistent trajectory.
- **Online processing** is maintained because only `window_size` frames need to reside in memory at once, while the overlap ensures continuity.

## Frequently Asked Questions

### What is the difference between anchor context and pose-reference windows?

The **anchor context** provides the initial global reference using the first `num_scale_frames` processed with bidirectional attention, establishing absolute scale and orientation. The **pose-reference window** handles subsequent frames by processing them in overlapping chunks and aligning each new window to the previous ones, correcting drift that accumulates after the anchor context ends.

### How does the similarity transform remove drift?

The similarity transform (scale **s**, rotation **R**, translation **t**) computed by `_pairwise_alignment` quantifies the coordinate system mismatch between consecutive windows. By applying this transform via `_warp_predictions`, the model warps the current window's predictions to match the global coordinate frame of the first window, effectively subtracting the accumulated error between windows.

### What happens if overlap_keyframes is set to zero?

Setting `overlap_keyframes` to zero removes the shared reference points between windows. Without overlapping frames, `_pairwise_alignment` cannot compute the similarity transform required to stitch windows together, causing the trajectory to break at window boundaries and allowing drift to accumulate unchecked.

### Can I use the pose-reference window with custom video lengths?

Yes. The `inference_windowed` method in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py) dynamically handles arbitrary sequence lengths by padding the final window and adjusting the overlap calculation. The drift correction mechanism operates independently of total video length, making it suitable for both short clips and long-form streaming applications.