How the Pose-Reference Window Enables Drift Correction in LingBot-Map

The pose-reference window in LingBot-Map corrects accumulated drift by splitting video streams into overlapping keyframe windows, comparing pose encodings between shared anchor frames, and warping predictions using a learned similarity transform to maintain a globally consistent coordinate system.

Robbyant/lingbot-map processes video streams in an online fashion, where sequential frame-by-frame estimation causes tiny pose errors to compound into severe drift over long sequences. The pose-reference window architecture solves this by periodically re-anchoring the scene using overlapping keyframes that bridge consecutive windows, allowing the model to align and stitch local trajectories into a single coherent global path.

The Drift Problem in Online Video Processing

When a model processes thousands of frames sequentially, each new pose prediction builds upon the previous estimate. This recursive dependency means that even sub-pixel errors accumulate, causing the camera trajectory to drift away from its true path. Traditional solutions require storing all frames in memory for global optimization, which violates streaming constraints. LingBot-Map solves this through a three-part architectural design that balances memory efficiency with global consistency.

Three-Pillar Architecture for Global Consistency

The repository implements drift correction through three coordinated components:

• Anchor context – Supplies the first num_scale_frames frames processed with bidirectional attention, establishing a solid global reference for scale and orientation.
• Pose-reference window – Splits the remaining sequence into overlapping windows of keyframes, using stored pose encodings to align and warp predictions between windows.
• Trajectory memory – Maintains a running record of aligned poses, enabling downstream modules like the point-cloud generator to operate on a globally consistent trajectory.

The pose-reference window serves as the active drift-correction mechanism, functioning like a sliding reset that stitches local window predictions into a unified global frame.

How the Pose-Reference Window Corrects Drift

Window Creation and Overlap Strategy

The windowing logic is implemented in inference_windowed (line 1025 in lingbot_map/models/gct_stream_window_v2.py). This method defines the streaming parameters that govern how the sequence is partitioned:

• window_size – Number of keyframes processed per window.
• overlap_keyframes – Shared frames between consecutive windows that serve as alignment anchors.
• keyframe_interval – Frequency of frames stored in the KV cache.

By maintaining overlapping keyframes between windows, the system creates common reference points that exist in both the previous and current coordinate frames.

Alignment via Pose Encoding Comparison

When a new window finishes processing, the model executes _pairwise_alignment (line 1029) to estimate the similarity transform between the previous and current windows. This function extracts the pose encodings (pose_enc) from the last overlapping keyframe of the previous window and the first overlapping keyframe of the current window.

The alignment computes a similarity transform comprising:

• Scale s
• Rotation R
• Translation t

These parameters quantify exactly how much the current window has drifted relative to the global coordinate system established by previous windows.

Warping and Stitching

After computing the alignment, _warp_predictions (line 1039) applies the similarity transform to all predictions in the current window—including pose, depth, and world points—effectively warping them into the coordinate frame of the first window.

The _align_and_stitch_windows method (line 1055) iterates over all windows, applies the pairwise alignment, and concatenates the results via _stitch_windows. This process removes accumulated drift between windows while preserving the local accuracy of windowed inference.

Implementation in Code

You can enable drift correction using the command-line interface or the Python API.

Command-Line Usage

Run the interactive demo in windowed mode to activate the pose-reference window:

python demo.py \
    --model_path /path/to/lingbot-map-long.pt \
    --image_folder example/university \
    --mode windowed \
    --window_size 128 \
    --overlap_keyframes 16 \
    --keyframe_interval 2 \
    --mask_sky

The --overlap_keyframes parameter ensures shared anchors exist between windows, which is essential for the drift correction mechanism to function.

Python API Usage

For programmatic access, use the GCTStream class with inference_windowed:

import torch
from lingbot_map.models.gct_stream_window_v2 import GCTStream

model = GCTStream().to('cuda')
images = torch.rand(1, 2000, 3, 518, 518).cuda()   # 2000-frame demo video

# Windowed inference with automatic drift correction

pred = model.inference_windowed(
    images,
    window_size=128,
    overlap_keyframes=16,
    keyframe_interval=2,
    output_device='cpu',
)

# Pose encodings are already drift-corrected

pose_enc = pred['pose_enc']   # shape [B, S, 9]

Inspecting Alignment Between Windows

To verify the drift correction magnitude between consecutive windows:

window_a = pred_a   # output dict from first window

window_b = pred_b   # output dict from second window

# Compute the similarity transform used for stitching

s, R, t = model._pairwise_alignment(
    window_a, 
    window_b,
    overlap=model._last_overlap_size,
    batch_size=window_a['pose_enc'].shape[0],
    device=window_a['pose_enc'].device,
    dtype=window_a['pose_enc'].dtype
)

print('Scale:', s.mean().item())
print('Rotation norm:', R.norm(p=2, dim=[1,2]).mean().item())
print('Translation norm:', t.norm(p=2, dim=1).mean().item())

Summary

• The pose-reference window combats drift by re-anchoring the scene every window_size frames using overlapping keyframes as shared reference points.
• Alignment occurs via _pairwise_alignment, which compares 9-dimensional pose encodings (pose_enc) between overlapping frames to estimate scale, rotation, and translation differences.
• Warping via _warp_predictions applies the similarity transform to all predictions, stitching windows into a globally consistent trajectory.
• Online processing is maintained because only window_size frames need to reside in memory at once, while the overlap ensures continuity.

Frequently Asked Questions

What is the difference between anchor context and pose-reference windows?

The anchor context provides the initial global reference using the first num_scale_frames processed with bidirectional attention, establishing absolute scale and orientation. The pose-reference window handles subsequent frames by processing them in overlapping chunks and aligning each new window to the previous ones, correcting drift that accumulates after the anchor context ends.

How does the similarity transform remove drift?

The similarity transform (scale s, rotation R, translation t) computed by _pairwise_alignment quantifies the coordinate system mismatch between consecutive windows. By applying this transform via _warp_predictions, the model warps the current window's predictions to match the global coordinate frame of the first window, effectively subtracting the accumulated error between windows.

What happens if overlap_keyframes is set to zero?

Setting overlap_keyframes to zero removes the shared reference points between windows. Without overlapping frames, _pairwise_alignment cannot compute the similarity transform required to stitch windows together, causing the trajectory to break at window boundaries and allowing drift to accumulate unchecked.

Can I use the pose-reference window with custom video lengths?

Yes. The inference_windowed method in lingbot_map/models/gct_stream_window_v2.py dynamically handles arbitrary sequence lengths by padding the final window and adjusting the overlap calculation. The drift correction mechanism operates independently of total video length, making it suitable for both short clips and long-form streaming applications.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}