# Overlap Keyframes for Cross-Window Pose Alignment in LingBot-Map

> Learn how overlap keyframes ensure pose alignment across windows in LingBot-Map, enabling robust scale estimation and rigid transform calculation for long video sequences.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: deep-dive
- Published: 2026-07-01

---

**Overlap keyframes guarantee that consecutive processing windows share at least one mutual keyframe, enabling robust scale estimation and rigid transform calculation to align poses across window boundaries in long video sequences.**

LingBot-Map processes extended video sequences by splitting them into independent windows to manage memory constraints and KV-cache limitations. Because each window initializes a fresh cache and operates in its own local coordinate frame, the library uses overlap keyframes to ensure reliable cross-window pose alignment and maintain global trajectory consistency across the entire sequence.

## The Challenge of Windowed Inference

When processing long videos, LingBot-Map splits frames into discrete **windows** to handle computational constraints. Each window processes independently with a fresh KV-cache, as implemented in the base model class. This isolation creates a critical problem at window boundaries: the pose predictions for the current window exist in a different coordinate frame than the previous window.

Without explicit alignment, the transition between windows produces discontinuities in the global trajectory. The model solves this by defining an **overlap region** containing the last frames of the previous window and the first frames of the current window. However, simply overlapping frames does not guarantee usable data for alignment.

## How Overlap Keyframes Guarantee Robust Alignment

Specifying overlap in raw frames (`overlap_size`) risks alignment failure when the `keyframe_interval` is large. If no overlapped frames happen to be keyframes in both windows, the system falls back to a degenerate "first-overlap-frame" strategy that compromises accuracy.

**Overlap keyframes** solve this by specifying the overlap in terms of keyframe count rather than frame count. In [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py), the `inference_windowed` method converts this to an actual frame count that ensures coverage:

```python
eff_overlap = max(ws, overlap_keyframes * keyframe_interval)

```

This calculation (lines 991-1002) guarantees that the overlap region contains at least one keyframe from each window, ensuring that `_pairwise_alignment` receives reliable depth correspondence for scale estimation.

## The Alignment Pipeline

The cross-window alignment operates in three stages, all implemented in [`gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream_window_v2.py):

### Finding Overlap Regions

The system identifies the overlapping frame indices between the current window and the previous window. Within this region, the model searches for frames marked as keyframes in **both** windows—these are the **overlap keyframes** that provide the geometric constraints for alignment.

### Pairwise Alignment via `_pairwise_alignment`

The core alignment logic resides in `_pairwise_alignment` (lines 398-424). This function accepts the overlap size and returns the transformation parameters needed to map the current window into the previous window's coordinate frame:

- **scale**: The relative scale factor between windows
- **R**: The rotation matrix
- **t**: The translation vector

The function uses only the paired keyframes within the overlap region, leveraging their reliable depth estimates to compute these parameters.

### Warping and Stitching

After computing the transformation, `_warp_predictions` applies the scale, rotation, and translation to every prediction in the current window. The `_stitch_windows` function then concatenates the warped predictions onto the global trajectory, producing a seamless transition across the window boundary.

## Configuring Overlap Keyframes in Practice

Use the `overlap_keyframes` parameter in `inference_windowed` to ensure robust alignment when processing long sequences:

```python

# Process a long sequence with explicit overlap expressed in keyframes

prediction = model.inference_windowed(
    images=video_tensor,          # [S, 3, H, W] or [B, S, 3, H, W]

    window_size=16,               # number of keyframes per window

    overlap_keyframes=4,          # ensure 4 overlapping keyframes

    keyframe_interval=2,          # every 2nd frame is a keyframe

    num_scale_frames=2,           # frames used for scale estimation

)

```

In this configuration:
- `overlap_keyframes=4` requests four mutual keyframes between windows
- With `keyframe_interval=2`, the system calculates an effective overlap of at least 8 frames
- `_pairwise_alignment` receives guaranteed keyframe pairs for accurate scale and pose estimation

## Summary

- **Windowed inference** processes long videos in isolated chunks with independent KV-caches, creating coordinate frame mismatches at boundaries.
- **Overlap keyframes** specify overlap in terms of keyframe count rather than frames, ensuring the overlap region contains usable geometric data.
- The conversion `eff_overlap = max(ws, overlap_keyframes * keyframe_interval)` in `inference_windowed` guarantees at least one paired keyframe for alignment.
- **`_pairwise_alignment`** computes scale, rotation, and translation using only these overlapping keyframes to align consecutive windows.
- **`_warp_predictions`** and **`_stitch_windows`** apply these transforms to maintain global trajectory consistency.

## Frequently Asked Questions

### What happens if no keyframes overlap between windows?

If the overlap region contains no mutual keyframes, the system falls back to a degenerate alignment strategy using the first overlap frame. This fallback is less robust and may produce inaccurate scale estimates or trajectory discontinuities, which is why `overlap_keyframes` is recommended over raw frame counts.

### How does `overlap_keyframes` differ from `overlap_size`?

`overlap_keyframes` specifies the minimum number of keyframes that must overlap between windows, while `overlap_size` specifies the overlap in raw frames. When using `overlap_keyframes`, LingBot-Map internally calculates the actual frame count needed to guarantee the requested number of keyframe pairs, accounting for the `keyframe_interval`.

### Why is scale estimation affected by keyframe pairs?

Scale estimation relies on geometric correspondence between frames with known depth. Keyframes provide reliable depth estimates and feature correspondences. Without overlapping keyframes, the system lacks sufficient geometric constraints to accurately compute the relative scale factor between windows, leading to drift in the global reconstruction.

### Which source files handle the alignment logic?

The primary implementation resides in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py), specifically the `inference_windowed` method and `_pairwise_alignment` function. Supporting utilities for pose encoding are found in [`lingbot_map/utils/pose_enc.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/pose_enc.py), while the base KV-cache management is implemented in [`lingbot_map/models/gct_base.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_base.py).