# Impact of keyframe_interval and window_size on Long-Sequence Inference Memory Usage in LingBot-Map

> Explore how keyframe_interval and window_size affect long-sequence inference memory usage in LingBotMap. Optimize GPU memory for video processing.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: performance
- Published: 2026-07-04

---

**`keyframe_interval` subsamples the KV cache by keeping only every N‑th frame, while `window_size` caps the absolute number of keyframe slots per inference chunk, together controlling peak GPU memory during long-video processing.**

LingBot-Map is an open-source 3D reconstruction system that processes video streams using transformer-based attention mechanisms. When running inference on videos exceeding the training sequence length of approximately 320 frames, **GPU memory consumption** becomes the primary bottleneck due to unbounded KV cache growth. The repository provides two critical configuration parameters—`keyframe_interval` and `window_size`—that directly control this memory footprint in the `GCTStream` class.

## Understanding the KV Cache Bottleneck

The transformer architecture in LingBot-Map maintains a **KV cache** (keys and values of the attention layers) that stores hidden states for every keyframe processed. As the sequence length grows, this cache expands linearly, eventually dominating GPU memory and triggering out-of-memory (OOM) errors on long videos (tens of thousands of frames). The implementation in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py) provides two complementary mechanisms to bound this growth: subsampling via `keyframe_interval` and chunking via `window_size`.

## How keyframe_interval Reduces Memory Consumption

The `keyframe_interval` parameter (CLI flag `--keyframe_interval`) determines how frequently frames are permanently cached. When set to a value greater than 1, the model discards KV entries for intermediate frames, keeping only every *N*‑th frame after the initial scale frames.

In the streaming loop at line 665 of [`gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream_window_v2.py), the logic checks:

```python
is_keyframe = (keyframe_interval <= 1) or ((i - scale_frames) % keyframe_interval == 0)

```

For non-keyframes, the code invokes `_set_skip_append(True)`, which prevents the transformer from appending to the KV cache. This reduces the cache size by roughly a factor of *N*. For example, setting `--keyframe_interval 13` on a 10,000-frame video results in only ~770 keyframes being cached instead of 10,000, dramatically lowering memory requirements.

## How window_size Caps Peak Memory

While `keyframe_interval` slows cache growth, `window_size` establishes a hard upper bound on the number of keyframe slots that exist simultaneously in GPU memory. This parameter (exposed as `--window_size` in [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py) and the `window_size` argument in `GCTStream.inference_windowed`) defines how many keyframes (including `num_scale_frames` that are always retained) are processed before the KV cache is explicitly reset.

The windowed inference mode processes blocks sequentially:

1.  Load `window_size` keyframes into a fresh KV cache.
2.  Generate predictions for that window.
3.  Stitch results with adjacent windows using overlap handling.
4.  Reset the cache and repeat.

Because the cache is cleared between windows, peak GPU memory is bounded by the `window_size` rather than the total video length. Larger windows improve temporal consistency but increase peak memory usage; smaller windows reduce memory at the cost of more frequent resets.

## Calculating Actual Frame Coverage

When combining both parameters, the effective number of **actual** video frames covered by a single window follows the formula documented at lines 1044–1048 in [`gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream_window_v2.py):

```text
actual_frames = num_scale_frames + (window_size - num_scale_frames) × keyframe_interval

```

This calculation is critical for planning batch sizes. For instance, with 4 scale frames, a window_size of 128, and a keyframe_interval of 13, one window covers 4 + (124 × 13) = 1,616 actual frames despite only caching 128 keyframes.

The overlap between consecutive windows must also account for the interval. At lines 1099–1102, the code converts `overlap_keyframes` into actual frame overlap using:

```python
eff_overlap = max(ws, overlap_keyframes * keyframe_interval)

```

This ensures that the overlapping region contains sufficient keyframes from both windows to maintain alignment quality during stitching.

## Practical Configuration Strategies

The repository recommendations in [`README.md`](https://github.com/Robbyant/lingbot-map/blob/main/README.md) (lines 24–28) provide concrete guidance for trading memory against temporal resolution:

- **Small `keyframe_interval` (e.g., 1–2)**: Every frame (or every second frame) becomes a keyframe. This preserves maximum temporal detail but causes rapid cache growth, suitable only for short sequences under 300 frames.

- **Large `keyframe_interval` (e.g., 13)**: Used for ultra-long sequences exceeding 3,000 frames. By caching only every 13th frame, the KV cache remains small enough to fit in consumer GPU memory even for 25,000-frame videos.

- **Small `window_size` (e.g., 32–64)**: Forces frequent cache resets, keeping peak memory low but increasing computational overhead for window stitching.

- **Large `window_size` (e.g., 128)**: Allows more keyframes per window, improving cross-window alignment but requiring more VRAM. The README suggests pairing `--window_size 128` with `--overlap_keyframes 8` for videos over 3,000 frames.

## Implementation Examples

For simple streaming inference on moderate-length sequences, use the CLI with a modest interval:

```bash
python demo.py \
    --model_path /path/to/lingbot-map-long.pt \
    --image_folder example/university \
    --keyframe_interval 2

```

For very long videos that would otherwise exhaust GPU memory, combine windowing with aggressive subsampling:

```bash
python demo.py \
    --model_path /path/to/lingbot-map-long.pt \
    --video_path indoor_walkthrough.mp4 \
    --mode windowed \
    --window_size 128 \
    --overlap_keyframes 8 \
    --keyframe_interval 13

```

For programmatic control, call the API directly with the same semantics:

```python
from lingbot_map.models.gct_stream_window_v2 import GCTStream

model = GCTStream(...)
preds = model.inference_windowed(
    images,                # [B, S, 3, H, W] tensor

    window_size=128,       # keyframe slots per window

    overlap_keyframes=8,   # overlap expressed in keyframes

    keyframe_interval=13,  # cache every 13th frame

    flow_threshold=0.0,    # fixed-interval mode

)

```

## Summary

- **`keyframe_interval`** subsamples the KV cache by retaining only every *N*‑th frame, reducing memory growth proportionally to the interval size as implemented at line 665 of [`gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream_window_v2.py).
- **`window_size`** caps the absolute number of keyframe entries in GPU memory by resetting the cache after each window, preventing OOM errors on arbitrarily long videos.
- The **actual frame coverage** per window scales multiplicatively with both parameters, calculated as `num_scale_frames + (window_size - num_scale_frames) × keyframe_interval`.
- **Overlap handling** converts keyframe-based overlap to frame-based overlap using the interval factor (lines 1099–1102) to ensure proper window stitching.
- According to the repository documentation, values of `--keyframe_interval 2` for moderate sequences and `--keyframe_interval 13` with `--window_size 128` for long sequences (>3000 frames) provide optimal memory-efficiency trade-offs.

## Frequently Asked Questions

### What happens if I set keyframe_interval to 1 on a 10,000-frame video?

Setting `keyframe_interval` to 1 forces the model to cache every single frame as a keyframe. On a 10,000-frame sequence, the KV cache will grow to store 10,000 sets of hidden states, likely exceeding GPU memory limits and causing an out-of-memory error unless you have substantial VRAM (typically >24GB) or use a very small `window_size` to periodically clear the cache.

### How does window_size interact with keyframe_interval?

The `window_size` parameter counts **keyframes**, not raw frames. Therefore, if `keyframe_interval` is set to 13, a `window_size` of 128 actually covers 128 cached keyframes representing roughly 1,600 actual video frames. The cache reset occurs after processing 128 keyframes, regardless of how many non-keyframes were skipped in between, meaning peak memory is bounded by the window size while temporal reach is extended by the interval.

### Why must overlap_keyframes account for keyframe_interval?

The stitching algorithm requires overlapping keyframes from adjacent windows to align predictions correctly. Since `keyframe_interval` determines the spacing between cached frames, the actual frame overlap must be calculated as `overlap_keyframes × keyframe_interval` (see lines 1099–1102 in [`gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream_window_v2.py)). If the overlap is too small relative to the interval, windows may fail to share common keyframes, causing temporal drift in the reconstruction.

### Can I use windowed inference without a keyframe_interval?

Yes, you can set `keyframe_interval=1` while using `inference_windowed` to process every frame but still benefit from the memory cap provided by `window_size`. This configuration provides the highest temporal resolution (no skipped frames) while preventing OOM errors through periodic cache resets. However, each window will process exactly `window_size` frames, requiring more frequent stitching overhead compared to using a larger interval.