# How the `overlap_keyframes` Parameter Ensures Stable Pose Alignment Across Sliding Windows

> Discover how overlap_keyframes ensures stable pose alignment by converting keyframe units to frame counts, guaranteeing robust similarity transforms between sliding windows.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: internals
- Published: 2026-07-31

---

**The `overlap_keyframes` parameter guarantees stable pose alignment by converting keyframe units into actual frame counts, ensuring every overlap region contains at least one mutually-present keyframe that enables computation of a robust similarity transform between windows.**

In the `lingbot-map` repository, windowed inference splits long video sequences into manageable chunks to process them independently. The `overlap_keyframes` parameter in [`lingbot_map/models/gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py) provides a high-level mechanism to specify overlap between consecutive sliding windows using keyframe units rather than raw frame counts, directly preventing pose drift when stitching predictions back together.

## The Alignment Challenge in Windowed Inference

When processing video sequences longer than the model's maximum window size, the system divides frames into overlapping segments. Each window generates independent pose predictions, and these must be aligned into a single global coordinate system. Stable alignment requires **sufficient overlap** between windows and **keyframe pairing**—specifically, the overlap region must contain at least one cached keyframe present in both the previous and current windows. Without this shared reference point, the similarity transform (scale and translation) computed between windows becomes unreliable, causing trajectory drift.

## How `overlap_keyframes` Converts to Effective Frame Overlap

The implementation in [`lingbot_map/models/gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py) (lines 31-40) handles the conversion from keyframe units to actual frames through a priority-based calculation that ensures robust alignment conditions.

### Parameter Priority and Conversion Logic

The system checks `overlap_keyframes` first, taking precedence over the legacy `overlap_size` parameter:

```python
if overlap_keyframes is not None:
    # Conversion logic executes here

elif overlap_size is not None:
    # Legacy frame-based overlap

else:
    # Default handling

```

The conversion multiplies the requested keyframe overlap by the `keyframe_interval` (the spacing between cached keyframes) to yield an **effective overlap** in actual frames:

```python
kf = max(keyframe_interval, 1)
eff_overlap = max(ws, overlap_keyframes * kf)

```

This formula ensures that requesting 4 keyframes with a `keyframe_interval` of 2 produces 8 actual frames of overlap, guaranteeing that the region spans enough raw frames to include the cached keyframes from both windows.

### Minimum Overlap Enforcement

The calculation enforces a floor using `max(ws, ...)` where `ws` represents the number of **scale frames**—frames used for depth-ratio scaling that are always present at the start of every window. This guarantees the overlap region always contains these critical reference frames regardless of the keyframe calculation.

### Sequence Boundary Clamping

The effective overlap is clamped to the sequence length to prevent invalid windows:

```python
eff_overlap = min(eff_overlap, S-1) if S > 1 else 0

```

Where `S` represents the total number of frames in the sequence.

## The Stitching Process and Keyframe Pairing

Once calculated, `eff_overlap` passes to `_stitch_windows(warped_windows, win_sz, overlap)`, which performs the actual alignment. The method locates **paired keyframes** inside the overlap region—keyframes present in both the tail of the previous window and the head of the current window. The keyframe-pair logic, found in the `if is_kf_prev is not None ...` block within `_stitch_windows`, computes a robust similarity transform by aggregating depth-scaled transformations across these matched pairs.

Because the overlap region always contains at least one mutually-present keyframe, the pose encoding from the previous window transforms reliably into the next window's coordinate system. The merged output includes `"alignment_mode": "scaled"` and retains per-window transforms, producing a smooth global trajectory without drift.

## Practical Usage Examples

Configure windowed inference with automatic overlap calculation:

```python
pred = model.inference_windowed(
    images,                 # [S,3,H,W] tensor

    window_size=16,         # 16 keyframes per window

    overlap_keyframes=4,    # 4 keyframes of overlap (converted to actual frames)

    keyframe_interval=2,    # cache every 2nd frame

)

```

Explicitly control the conversion while disabling legacy parameters:

```python
pred = model.inference_windowed(
    images,
    window_size=16,
    overlap_keyframes=3,    # 3 keyframes → 3 * 2 = 6 actual frames overlap

    keyframe_interval=2,
    overlap_size=None,      # disable legacy frame-based overlap

)

```

Both calls compute `eff_overlap = max(ws, overlap_keyframes * keyframe_interval)`, ensuring the overlap region always includes at least one paired keyframe for robust alignment according to the implementation in [`lingbot_map/models/gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py).

## Summary

- **`overlap_keyframes`** specifies overlap in keyframe units rather than raw frames, taking precedence over `overlap_size` in [`lingbot_map/models/gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py).
- **Conversion formula** multiplies keyframes by `keyframe_interval` and enforces a minimum of `ws` scale frames: `eff_overlap = max(ws, overlap_keyframes * kf)`.
- **Keyframe pairing** requires the overlap region to contain at least one cached keyframe present in both consecutive windows to compute a reliable similarity transform.
- **Drift prevention** occurs because the guaranteed keyframe overlap provides stable reference points for `_stitch_windows` to align pose encodings between independent window predictions.

## Frequently Asked Questions

### What happens if `overlap_keyframes` is set to `None`?

When `overlap_keyframes` is `None`, the implementation falls back to checking `overlap_size` according to the parameter priority logic in [`lingbot_map/models/gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py). If both are `None`, the system uses default overlap calculations that may not guarantee keyframe presence in the overlap region, potentially reducing alignment stability.

### How does `overlap_keyframes` differ from `overlap_size`?

**`overlap_keyframes`** measures overlap in units of cached keyframes and automatically converts to actual frames using the `keyframe_interval`, while **`overlap_size`** specifies the overlap directly in raw frame counts without considering keyframe spacing. The keyframe-based approach ensures the overlap region always contains reference frames suitable for computing similarity transforms, whereas raw frame counts might miss keyframes entirely depending on the interval setting.

### Why must the overlap region contain at least one keyframe from each window?

The stitching algorithm in `_stitch_windows` computes a similarity transform (scale and translation) to align pose encodings between windows. This calculation requires corresponding reference points that exist in both coordinate systems. Keyframes serve as these anchors because their hidden states are cached and their positions are known in both the previous window's tail and the current window's head. Without a mutually-present keyframe, the system lacks correspondences to solve for the alignment transformation reliably.

### What are scale frames (`ws`) and why enforce minimum overlap with them?

Scale frames are specific frames used for depth-ratio scaling that appear at the start of every window. The implementation enforces `eff_overlap = max(ws, ...)` because these frames provide critical geometric information for normalizing depth predictions across windows. Ensuring they remain within the overlap region guarantees that the stitching process has access to consistent scale references when merging window predictions, preventing scale drift in the global trajectory.