How to Configure window_size and overlap_keyframes for Windowed Inference on Long Sequences in LingBot-Map

Set window_size to define the number of keyframes processed per GPU-bound window and overlap_keyframes to ensure temporal continuity across windows, where the effective overlap equals max(num_scale_frames, overlap_keyframes * keyframe_interval).

LingBot-Map processes very long video sequences by splitting them into windows that are processed independently with a fresh KV-cache. This keeps GPU memory bounded while still allowing high-quality predictions across the whole sequence. Configuring window_size and overlap_keyframes correctly in the Robbyant/lingbot-map repository is essential for balancing memory constraints against temporal consistency in the reconstructed 3-D map.

Understanding Windowed Inference in LingBot-Map

The core of this mechanism lives in lingbot_map/models/gct_stream_window_v2.py — specifically the inference_windowed method at line 1025. When processing sequences longer than what fits in GPU memory, the model divides the input into segments rather than processing the entire video at once.

What window_size Controls

window_size counts keyframes, not raw frames. A window contains a set of scale frames processed with bidirectional attention (defaulting to self.num_frame_for_scale), plus the keyframes whose KV-cache is retained.

The actual number of frames processed in a window follows this formula:

actual_frames = scale_frames + (window_size - scale_frames) * keyframe_interval

where keyframe_interval controls how often a frame is stored as a keyframe (lines 1043-1048 in gct_stream_window_v2.py).

Increasing window_size gives the model more keyframes per window, improving quality at the cost of higher memory usage. Decreasing it reduces memory but may degrade temporal consistency between windows.

The Role of overlap_keyframes

By default, overlap between consecutive windows is expressed in actual frames via the overlap_size argument (line 1053). Supplying overlap_keyframes switches the overlap to be expressed in keyframes instead (lines 1055-1058).

This parameter is particularly valuable when keyframe_interval > 1, because it guarantees that at least one keyframe appears in the overlapping region. This is required for reliable cross-window alignment and temporal consistency.

How Overlap Calculation Works Internally

When overlap_keyframes is provided, the library converts the keyframe overlap to actual frames using:

effective_overlap = max(num_scale_frames, overlap_keyframes * keyframe_interval)

The code then clamps this value to at most the current window size (lines 1099-1102). This ensures the next window's scale phase always has enough frames to function correctly.

overlap_keyframes takes precedence over overlap_size; if both are supplied, the latter is ignored. The effective overlap is always at least the number of scale frames, ensuring the bidirectional attention mechanism works correctly across window boundaries.

Configuration Guidelines for Different Scenarios

Use these settings as starting points based on your video length and GPU memory:

• Very long sequences (≫ 1k frames): Use small window_size (e.g., 8-12) to keep per-window memory low; set overlap_keyframes approximately equal to window_size to guarantee overlap.
• High-quality temporal consistency: Select larger window_size (e.g., 16-24) and overlap_keyframes approximately window_size // 2.
• Fast inference on limited GPU: Reduce window_size and rely on the default overlap_size (which equals the number of scale frames).
• Non-uniform motion (fast camera moves): Enable flow-based keyframe detection (flow_threshold>0) and keep overlap_keyframes modest (e.g., 2-4). The flow logic creates extra windows where needed.

Code Examples

Programmatic Configuration in Python

Configure windowed inference directly through the inference_windowed method:

import torch
from lingbot_map.models.gct_stream_window_v2 import GCTStream

# Load a pretrained model

model = GCTStream(pretrained_path='path/to/checkpoint.pt')
model.eval()
model.to('cuda')

# Dummy long video: 2000 frames, 3×256×256 RGB

images = torch.rand(1, 2000, 3, 256, 256, device='cuda')

# Example 1: Small windows with keyframe overlap

preds = model.inference_windowed(
    images,
    window_size=12,               # 12 keyframes per window

    overlap_keyframes=6,          # 6 keyframes overlap → 12 actual frames (since keyframe_interval=1)

    keyframe_interval=1,          # Store every frame as a keyframe

    num_scale_frames=4,           # Use 4 scale frames per window

)

# Example 2: Larger keyframe interval with explicit overlap

preds = model.inference_windowed(
    images,
    window_size=16,
    keyframe_interval=4,          # Keep KV cache for every 4th frame

    overlap_keyframes=4,          # Overlap = 4 keyframes → 16 actual frames (4 × interval)

    num_scale_frames=4,
)

Command-Line Interface Usage

The demo.py script exposes these parameters through command-line arguments (lines 362-400):

python demo.py path/to/video.mp4 \
    --window-size 12 \
    --overlap-keyframes 6 \
    --keyframe-interval 1 \
    --num-scale-frames 4

These arguments map directly to the inference_windowed method parameters.

Summary

• window_size defines keyframe count, not raw frame count, controlling GPU memory per window through the formula actual_frames = scale_frames + (window_size - scale_frames) * keyframe_interval.
• overlap_keyframes specifies overlap in keyframes rather than frames, converting to effective_overlap = max(num_scale_frames, overlap_keyframes * keyframe_interval).
• overlap_keyframes takes precedence over overlap_size when both are provided.
• The effective overlap is always clamped to at least the number of scale frames to ensure proper bidirectional attention across windows.
• Adjust these parameters based on sequence length and GPU memory, using smaller windows with substantial overlap for very long sequences.

Frequently Asked Questions

What is the difference between window_size and overlap_keyframes?

window_size controls how many keyframes are processed in each independent window, directly affecting GPU memory usage. overlap_keyframes determines how many keyframes are shared between consecutive windows to maintain temporal consistency. While window_size defines the processing unit, overlap_keyframes ensures continuity across window boundaries by guaranteeing shared keyframes in the overlap region.

How does keyframe_interval affect the actual window size?

The keyframe_interval parameter controls how frequently frames are stored as keyframes. The actual frame count per window follows actual_frames = scale_frames + (window_size - scale_frames) * keyframe_interval. When keyframe_interval is greater than 1, the actual number of frames processed exceeds the window_size value, as window_size counts only keyframes, not intermediate frames.

Why should I use overlap_keyframes instead of overlap_size?

Use overlap_keyframes when keyframe_interval > 1 to ensure that at least one keyframe appears in the overlapping region between windows. This is essential for reliable cross-window alignment because the KV-cache synchronization depends on shared keyframes. The default overlap_size (measured in actual frames) might not guarantee keyframe presence in the overlap when frames are subsampled.

Where is the windowed inference logic implemented in the source code?

The windowed inference mechanism is implemented in lingbot_map/models/gct_stream_window_v2.py within the inference_windowed method starting at line 1025. The overlap calculation logic resides at lines 1099-1102, while parameter parsing for the CLI interface appears in demo.py at lines 362-400.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}