Impact of keyframe_interval and window_size on Long-Sequence Inference Memory Usage in LingBot-Map

keyframe_interval subsamples the KV cache by keeping only every N‑th frame, while window_size caps the absolute number of keyframe slots per inference chunk, together controlling peak GPU memory during long-video processing.

LingBot-Map is an open-source 3D reconstruction system that processes video streams using transformer-based attention mechanisms. When running inference on videos exceeding the training sequence length of approximately 320 frames, GPU memory consumption becomes the primary bottleneck due to unbounded KV cache growth. The repository provides two critical configuration parameters—keyframe_interval and window_size—that directly control this memory footprint in the GCTStream class.

Understanding the KV Cache Bottleneck

The transformer architecture in LingBot-Map maintains a KV cache (keys and values of the attention layers) that stores hidden states for every keyframe processed. As the sequence length grows, this cache expands linearly, eventually dominating GPU memory and triggering out-of-memory (OOM) errors on long videos (tens of thousands of frames). The implementation in lingbot_map/models/gct_stream_window_v2.py provides two complementary mechanisms to bound this growth: subsampling via keyframe_interval and chunking via window_size.

How keyframe_interval Reduces Memory Consumption

The keyframe_interval parameter (CLI flag --keyframe_interval) determines how frequently frames are permanently cached. When set to a value greater than 1, the model discards KV entries for intermediate frames, keeping only every N‑th frame after the initial scale frames.

In the streaming loop at line 665 of gct_stream_window_v2.py, the logic checks:

is_keyframe = (keyframe_interval <= 1) or ((i - scale_frames) % keyframe_interval == 0)

For non-keyframes, the code invokes _set_skip_append(True), which prevents the transformer from appending to the KV cache. This reduces the cache size by roughly a factor of N. For example, setting --keyframe_interval 13 on a 10,000-frame video results in only ~770 keyframes being cached instead of 10,000, dramatically lowering memory requirements.

How window_size Caps Peak Memory

While keyframe_interval slows cache growth, window_size establishes a hard upper bound on the number of keyframe slots that exist simultaneously in GPU memory. This parameter (exposed as --window_size in demo.py and the window_size argument in GCTStream.inference_windowed) defines how many keyframes (including num_scale_frames that are always retained) are processed before the KV cache is explicitly reset.

The windowed inference mode processes blocks sequentially:

1. Load window_size keyframes into a fresh KV cache.
2. Generate predictions for that window.
3. Stitch results with adjacent windows using overlap handling.
4. Reset the cache and repeat.

Because the cache is cleared between windows, peak GPU memory is bounded by the window_size rather than the total video length. Larger windows improve temporal consistency but increase peak memory usage; smaller windows reduce memory at the cost of more frequent resets.

Calculating Actual Frame Coverage

When combining both parameters, the effective number of actual video frames covered by a single window follows the formula documented at lines 1044–1048 in gct_stream_window_v2.py:

actual_frames = num_scale_frames + (window_size - num_scale_frames) × keyframe_interval

This calculation is critical for planning batch sizes. For instance, with 4 scale frames, a window_size of 128, and a keyframe_interval of 13, one window covers 4 + (124 × 13) = 1,616 actual frames despite only caching 128 keyframes.

The overlap between consecutive windows must also account for the interval. At lines 1099–1102, the code converts overlap_keyframes into actual frame overlap using:

eff_overlap = max(ws, overlap_keyframes * keyframe_interval)

This ensures that the overlapping region contains sufficient keyframes from both windows to maintain alignment quality during stitching.

Practical Configuration Strategies

The repository recommendations in README.md (lines 24–28) provide concrete guidance for trading memory against temporal resolution:

• Small keyframe_interval (e.g., 1–2): Every frame (or every second frame) becomes a keyframe. This preserves maximum temporal detail but causes rapid cache growth, suitable only for short sequences under 300 frames.

• Large keyframe_interval (e.g., 13): Used for ultra-long sequences exceeding 3,000 frames. By caching only every 13th frame, the KV cache remains small enough to fit in consumer GPU memory even for 25,000-frame videos.

• Small window_size (e.g., 32–64): Forces frequent cache resets, keeping peak memory low but increasing computational overhead for window stitching.

• Large window_size (e.g., 128): Allows more keyframes per window, improving cross-window alignment but requiring more VRAM. The README suggests pairing --window_size 128 with --overlap_keyframes 8 for videos over 3,000 frames.

Implementation Examples

For simple streaming inference on moderate-length sequences, use the CLI with a modest interval:

python demo.py \
    --model_path /path/to/lingbot-map-long.pt \
    --image_folder example/university \
    --keyframe_interval 2

For very long videos that would otherwise exhaust GPU memory, combine windowing with aggressive subsampling:

python demo.py \
    --model_path /path/to/lingbot-map-long.pt \
    --video_path indoor_walkthrough.mp4 \
    --mode windowed \
    --window_size 128 \
    --overlap_keyframes 8 \
    --keyframe_interval 13

For programmatic control, call the API directly with the same semantics:

from lingbot_map.models.gct_stream_window_v2 import GCTStream

model = GCTStream(...)
preds = model.inference_windowed(
    images,                # [B, S, 3, H, W] tensor

    window_size=128,       # keyframe slots per window

    overlap_keyframes=8,   # overlap expressed in keyframes

    keyframe_interval=13,  # cache every 13th frame

    flow_threshold=0.0,    # fixed-interval mode

)

Summary

• keyframe_interval subsamples the KV cache by retaining only every N‑th frame, reducing memory growth proportionally to the interval size as implemented at line 665 of gct_stream_window_v2.py.
• window_size caps the absolute number of keyframe entries in GPU memory by resetting the cache after each window, preventing OOM errors on arbitrarily long videos.
• The actual frame coverage per window scales multiplicatively with both parameters, calculated as num_scale_frames + (window_size - num_scale_frames) × keyframe_interval.
• Overlap handling converts keyframe-based overlap to frame-based overlap using the interval factor (lines 1099–1102) to ensure proper window stitching.
• According to the repository documentation, values of --keyframe_interval 2 for moderate sequences and --keyframe_interval 13 with --window_size 128 for long sequences (>3000 frames) provide optimal memory-efficiency trade-offs.

Frequently Asked Questions

What happens if I set keyframe_interval to 1 on a 10,000-frame video?

Setting keyframe_interval to 1 forces the model to cache every single frame as a keyframe. On a 10,000-frame sequence, the KV cache will grow to store 10,000 sets of hidden states, likely exceeding GPU memory limits and causing an out-of-memory error unless you have substantial VRAM (typically >24GB) or use a very small window_size to periodically clear the cache.

How does window_size interact with keyframe_interval?

The window_size parameter counts keyframes, not raw frames. Therefore, if keyframe_interval is set to 13, a window_size of 128 actually covers 128 cached keyframes representing roughly 1,600 actual video frames. The cache reset occurs after processing 128 keyframes, regardless of how many non-keyframes were skipped in between, meaning peak memory is bounded by the window size while temporal reach is extended by the interval.

Why must overlap_keyframes account for keyframe_interval?

The stitching algorithm requires overlapping keyframes from adjacent windows to align predictions correctly. Since keyframe_interval determines the spacing between cached frames, the actual frame overlap must be calculated as overlap_keyframes × keyframe_interval (see lines 1099–1102 in gct_stream_window_v2.py). If the overlap is too small relative to the interval, windows may fail to share common keyframes, causing temporal drift in the reconstruction.

Can I use windowed inference without a keyframe_interval?

Yes, you can set keyframe_interval=1 while using inference_windowed to process every frame but still benefit from the memory cap provided by window_size. This configuration provides the highest temporal resolution (no skipped frames) while preventing OOM errors through periodic cache resets. However, each window will process exactly window_size frames, requiring more frequent stitching overhead compared to using a larger interval.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}