Overlap Keyframes for Cross-Window Pose Alignment in LingBot-Map

Overlap keyframes guarantee that consecutive processing windows share at least one mutual keyframe, enabling robust scale estimation and rigid transform calculation to align poses across window boundaries in long video sequences.

LingBot-Map processes extended video sequences by splitting them into independent windows to manage memory constraints and KV-cache limitations. Because each window initializes a fresh cache and operates in its own local coordinate frame, the library uses overlap keyframes to ensure reliable cross-window pose alignment and maintain global trajectory consistency across the entire sequence.

The Challenge of Windowed Inference

When processing long videos, LingBot-Map splits frames into discrete windows to handle computational constraints. Each window processes independently with a fresh KV-cache, as implemented in the base model class. This isolation creates a critical problem at window boundaries: the pose predictions for the current window exist in a different coordinate frame than the previous window.

Without explicit alignment, the transition between windows produces discontinuities in the global trajectory. The model solves this by defining an overlap region containing the last frames of the previous window and the first frames of the current window. However, simply overlapping frames does not guarantee usable data for alignment.

How Overlap Keyframes Guarantee Robust Alignment

Specifying overlap in raw frames (overlap_size) risks alignment failure when the keyframe_interval is large. If no overlapped frames happen to be keyframes in both windows, the system falls back to a degenerate "first-overlap-frame" strategy that compromises accuracy.

Overlap keyframes solve this by specifying the overlap in terms of keyframe count rather than frame count. In lingbot_map/models/gct_stream_window_v2.py, the inference_windowed method converts this to an actual frame count that ensures coverage:

eff_overlap = max(ws, overlap_keyframes * keyframe_interval)

This calculation (lines 991-1002) guarantees that the overlap region contains at least one keyframe from each window, ensuring that _pairwise_alignment receives reliable depth correspondence for scale estimation.

The Alignment Pipeline

The cross-window alignment operates in three stages, all implemented in gct_stream_window_v2.py:

Finding Overlap Regions

The system identifies the overlapping frame indices between the current window and the previous window. Within this region, the model searches for frames marked as keyframes in both windows—these are the overlap keyframes that provide the geometric constraints for alignment.

Pairwise Alignment via _pairwise_alignment

The core alignment logic resides in _pairwise_alignment (lines 398-424). This function accepts the overlap size and returns the transformation parameters needed to map the current window into the previous window's coordinate frame:

• scale: The relative scale factor between windows
• R: The rotation matrix
• t: The translation vector

The function uses only the paired keyframes within the overlap region, leveraging their reliable depth estimates to compute these parameters.

Warping and Stitching

After computing the transformation, _warp_predictions applies the scale, rotation, and translation to every prediction in the current window. The _stitch_windows function then concatenates the warped predictions onto the global trajectory, producing a seamless transition across the window boundary.

Configuring Overlap Keyframes in Practice

Use the overlap_keyframes parameter in inference_windowed to ensure robust alignment when processing long sequences:

# Process a long sequence with explicit overlap expressed in keyframes

prediction = model.inference_windowed(
    images=video_tensor,          # [S, 3, H, W] or [B, S, 3, H, W]

    window_size=16,               # number of keyframes per window

    overlap_keyframes=4,          # ensure 4 overlapping keyframes

    keyframe_interval=2,          # every 2nd frame is a keyframe

    num_scale_frames=2,           # frames used for scale estimation

)

In this configuration:

• overlap_keyframes=4 requests four mutual keyframes between windows
• With keyframe_interval=2, the system calculates an effective overlap of at least 8 frames
• _pairwise_alignment receives guaranteed keyframe pairs for accurate scale and pose estimation

Summary

• Windowed inference processes long videos in isolated chunks with independent KV-caches, creating coordinate frame mismatches at boundaries.
• Overlap keyframes specify overlap in terms of keyframe count rather than frames, ensuring the overlap region contains usable geometric data.
• The conversion eff_overlap = max(ws, overlap_keyframes * keyframe_interval) in inference_windowed guarantees at least one paired keyframe for alignment.
• _pairwise_alignment computes scale, rotation, and translation using only these overlapping keyframes to align consecutive windows.
• _warp_predictions and _stitch_windows apply these transforms to maintain global trajectory consistency.

Frequently Asked Questions

What happens if no keyframes overlap between windows?

If the overlap region contains no mutual keyframes, the system falls back to a degenerate alignment strategy using the first overlap frame. This fallback is less robust and may produce inaccurate scale estimates or trajectory discontinuities, which is why overlap_keyframes is recommended over raw frame counts.

How does overlap_keyframes differ from overlap_size?

overlap_keyframes specifies the minimum number of keyframes that must overlap between windows, while overlap_size specifies the overlap in raw frames. When using overlap_keyframes, LingBot-Map internally calculates the actual frame count needed to guarantee the requested number of keyframe pairs, accounting for the keyframe_interval.

Why is scale estimation affected by keyframe pairs?

Scale estimation relies on geometric correspondence between frames with known depth. Keyframes provide reliable depth estimates and feature correspondences. Without overlapping keyframes, the system lacks sufficient geometric constraints to accurately compute the relative scale factor between windows, leading to drift in the global reconstruction.

Which source files handle the alignment logic?

The primary implementation resides in lingbot_map/models/gct_stream_window_v2.py, specifically the inference_windowed method and _pairwise_alignment function. Supporting utilities for pose encoding are found in lingbot_map/utils/pose_enc.py, while the base KV-cache management is implemented in lingbot_map/models/gct_base.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}