How Windowed Inference Manages Overlapping Keyframes for Cross-Window Pose Alignment in LingBot-Map

Windowed inference maintains global trajectory consistency by detecting paired keyframes in overlapping regions between consecutive windows, computing a similarity transform (scale, rotation, and translation) from these shared anchors, and warping subsequent windows into the coordinate system of the previous window before stitching.

LingBot-Map processes long video streams by dividing them into independent inference windows, each initialized with a fresh KV cache. To reconstruct a globally consistent camera trajectory from these disconnected segments, the repository implements a sophisticated alignment pipeline that leverages overlapping keyframes as spatial anchors. This article examines the implementation in lingbot_map/models/gct_stream_window_v2.py to explain how cross-window pose alignment transforms locally predicted coordinates into a unified global frame.

Defining the Overlap Region of Consecutive Windows

Before processing begins, the inference_windowed method establishes how many frames will be shared between adjacent windows. At lines 1089–1096, the code converts the user-supplied overlap_keyframes parameter into an effective frame overlap (eff_overlap) that is at least the size of the scale phase (ws). This ensures that sufficient temporal context exists for robust alignment regardless of the keyframe sampling interval.

The overlap calculation accounts for the fact that windows are processed independently with fresh KV caches. By enforcing a minimum overlap that satisfies the scale phase requirement, the system guarantees that geometric information from the end of one window remains available for comparison with the beginning of the next.

Detecting Paired Keyframes for Robust Alignment

The core of the alignment logic resides in _pairwise_alignment (lines 825–863). This function searches the overlapping region for frames that are marked as keyframes in both the previous window and the current window. The boolean tensor is_keyframe identifies which frames carry persistent representations in the KV cache.

If a paired keyframe exists, it serves as the anchor for computing the relative transformation. If no such pair exists, the algorithm falls back to using the first overlapping frame. This design prioritizes stable pose encodings—only cached keyframes retain reliable geometric representations, whereas transient non-keyframe states may exhibit drift.

Estimating the Similarity Transform from Depth Ratios

Once a paired keyframe is identified, the system computes the similarity transform that aligns the current window to the previous window. At lines 866–894 inside _pairwise_alignment, the code calculates:

• Rotation (R_ab): Derived from the relative orientation between the paired keyframes.
• Scale (s_ab): Computed by the helper _depth_ratio_scale, which compares depth ratios between corresponding points in the overlapping region.
• Translation (t_ab): Determined from the positional offset between the two frames.

This similarity transform accounts for scale ambiguities that arise because each window is processed independently without access to global metric information from adjacent segments.

Warping Predictions Across Coordinate Systems

With the similarity transform (s, R, t) computed, the _warp_predictions function (lines 901–954) applies this transformation to every prediction in the current window. The warping process updates:

• Pose encodings: Rotated and translated to match the previous window's coordinate frame.
• Depth maps: Scaled by the estimated factor s_ab.
• 3-D point clouds: Transformed according to the full similarity matrix.

This step effectively re-renders the current window's geometry as if it were observed from the same global reference frame as the preceding window, eliminating discontinuities at window boundaries.

Stitching Windows and Deduplicating Overlaps

After all windows are warped, the _stitch_windows function (lines 740–792) concatenates the temporally-indexed tensors into a single global trajectory. For each keyframe-type entry, the function constructs a slice table that drops the overlapping portion of every non-final window using effective_end = total - overlap.

Non-temporal entries, such as camera intrinsics, simply retain the latest value from the most recent window. This deduplication strategy ensures that the final output contains exactly one set of predictions per frame while preserving the geometric continuity established by the alignment transforms.

Retrieving Alignment Metadata

The inference_windowed method returns optional alignment metadata that enables post-hoc analysis of the stitching process. At lines 1014–1018 and 1022–1034, the code populates chunk_scales and chunk_transforms tensors containing the per-window similarity parameters. These tensors have shapes [B, N_windows, 1] and [B, N_windows, 4, 4] respectively, providing a complete record of how each window was rotated, scaled, and translated to fit the global trajectory.

Practical Implementation Examples

Basic Windowed Inference with Default Keyframe Interval

import torch
from lingbot_map.models.gct_stream_window_v2 import GCTStream

model = GCTStream(enable_camera=True, enable_point=True, enable_depth=True)
images = torch.rand(1, 120, 3, 518, 518)   # [B, S, C, H, W] in [0, 1]

# Run windowed inference with a window of 16 keyframes and default overlap.

pred = model.inference_windowed(
    images,
    window_size=16,          # number of keyframes per window

    overlap_keyframes=4,     # overlap expressed in keyframes

    keyframe_interval=2,     # store every 2nd frame in KV cache

)

This call triggers the overlap handling at lines 1089–1096 and the cross-window alignment logic in _pairwise_alignment (lines 825–894).

Flow-Based Keyframe Selection with Overlapping Keyframes

pred = model.inference_windowed(
    images,
    window_size=12,
    overlap_keyframes=3,
    flow_threshold=2.5,          # enable flow-based keyframe mode

    max_non_keyframe_gap=10,     # force a keyframe after 10 non-keyframes

)

When flow_threshold > 0, the implementation uses the flow-based branch (lines 1068–1084) while respecting the overlap-keyframe logic for alignment.

Inspecting the Alignment Metadata

# After inference, the per-window scale and transform tensors are available:

scales = pred["chunk_scales"]          # [B, N_windows, 1]

transforms = pred["chunk_transforms"]  # [B, N_windows, 4, 4]

print(scales.shape, transforms.shape)

The metadata is populated in _align_and_stitch_windows (lines 1014–1018).

Key Source Files in the Alignment Pipeline

• lingbot_map/models/gct_stream_window_v2.py: Contains inference_windowed, _pairwise_alignment, _warp_predictions, and _stitch_windows, implementing the complete windowed inference and cross-window alignment workflow.
• lingbot_map/utils/pose_enc.py: Provides utilities to convert pose encodings to extrinsics and intrinsics, used during flow magnitude computation and similarity transform calculation.
• lingbot_map/utils/geometry.py: Implements closed_form_inverse_se3 for converting between camera-to-world and world-to-camera matrices required during alignment.
• lingbot_map/utils/rotation.py: Offers quat_to_mat and mat_to_quat functions for handling rotations within pose encodings.

Summary

• Windowed inference splits long sequences into independent chunks with fresh KV caches, requiring explicit alignment to maintain global consistency.
• The system enforces a minimum overlap (eff_overlap) derived from overlap_keyframes and the scale phase size to ensure sufficient geometric context.
• Paired keyframe detection in _pairwise_alignment identifies shared anchors between windows, falling back to the first overlapping frame if necessary.
• A similarity transform (scale, rotation, translation) is computed from depth ratios and relative poses, then applied via _warp_predictions to unify coordinate systems.
• Window stitching in _stitch_windows deduplicates overlapping regions by slicing tensors at effective_end = total - overlap, producing a seamless global trajectory.

Frequently Asked Questions

Why are keyframes specifically required in the overlap region for alignment?

Keyframes are the only frames stored in the KV cache with persistent representations. By ensuring that at least one keyframe exists in both the previous and current window within the overlap region, the alignment can rely on stable pose encodings rather than transient, non-cached frames that may exhibit higher drift or noise.

What happens if no paired keyframe exists in the overlapping region?

If _pairwise_alignment cannot find a frame where is_keyframe is True for both windows simultaneously, it falls back to using the first overlapping frame as the alignment anchor. While this maintains continuity, paired keyframes provide superior stability because they carry confirmed geometric representations from both windows.

How does the similarity transform handle scale differences between independently processed windows?

The similarity transform includes a scale factor s_ab computed by _depth_ratio_scale, which compares depth ratios between corresponding points in the paired keyframes. This scale factor resolves the metric ambiguity that arises because each window is processed independently without global scale constraints, allowing the warping operation to harmonize the size of 3-D structures across window boundaries.

What is the difference between overlap_keyframes and eff_overlap in the configuration?

overlap_keyframes is the user-specified parameter indicating how many keyframes should overlap between windows. eff_overlap is the computed actual frame overlap used internally, which is guaranteed to be at least the size of the scale phase (ws). This conversion at lines 1089–1096 ensures that the overlap satisfies the minimum geometric requirements for reliable depth and pose estimation regardless of the keyframe sampling rate.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}