How to Configure the Keyframe Interval for Managing KV Cache Memory in LingBot-Map

Set keyframe_interval to a value greater than 1 in inference_streaming() or via the --keyframe_interval CLI flag to retain only every N-th frame in the transformer KV cache, reducing GPU memory usage by roughly a factor of N.

LingBot-Map implements a transformer-style KV cache to store attention keys and values for processed video frames. By default, every frame is treated as a keyframe (keyframe_interval = 1), which causes the cache to grow linearly and quickly exhaust GPU memory during long video sequences. Configuring the keyframe interval allows you to process thousands of frames while keeping memory consumption bounded.

Understanding the Keyframe Interval Mechanism

Default Behavior (Interval = 1)

When keyframe_interval is set to 1 (the default), LingBot-Map stores every processed frame in the KV cache. While this maximizes temporal coherence for pose estimation, it creates a memory bottleneck for long videos. The cache size grows proportionally with sequence length, and because the RoPE (Rotary Position Embedding) training data only covers approximately 320 frames, quality degrades beyond this point.

Memory Reduction with N > 1

Setting keyframe_interval to any integer N > 1 changes the caching strategy to retain only every N-th frame. Non-keyframe frames still generate full predictions, but their KV entries are immediately discarded. This reduces cache growth by roughly a factor of N, enabling inference over sequences containing thousands of frames without exceeding GPU memory limits.

The interval is applied after the initial scale frames (the first num_scale_frames that are processed together). According to the source code in lingbot_map/models/gct_stream_window_v2.py, the keyframe decision logic occurs at lines 64-66:

is_keyframe = (keyframe_interval <= 1) or ((i - scale_frames) % keyframe_interval == 0)

When is_keyframe evaluates to False, the model invokes _set_skip_append(True) at lines 68-70 to temporarily disable KV cache appending, keeping the cache size constant for that frame.

Configuring the Keyframe Interval

Command-Line Interface (CLI)

The recommended approach for benchmarking or batch processing is using the provided scripts. The argument is parsed in parse_args within scripts/benchmark_gct_memory.py (around line 64).

python -m scripts.benchmark_gct_memory \
    --device cuda:0 \
    --keyframe_interval 4 \
    --frame_counts 1000 2000 \
    --num_scale_frames 8

This configuration processes the initial 8 scale frames, then retains only every 4th subsequent frame in the KV cache.

Programmatic API

When integrating LingBot-Map into your own pipeline, pass keyframe_interval directly to the inference_streaming method in GCTStreamWindowV2. The default value is defined at lines 18-19 of gct_stream_window_v2.py.

from lingbot_map.models.gct_stream_window_v2 import GCTStreamWindowV2

model = GCTStreamWindowV2(...)

outputs = model.inference_streaming(
    images,                # Tensor shape: [B, S, 3, H, W]

    num_scale_frames=8,
    keyframe_interval=5,   # Retain every 5th frame in KV cache

    flow_threshold=0.0,    # Disable flow-based keyframe selection

)

Windowed Inference Mode

For extremely long videos, combine --keyframe_interval with --mode windowed. In this configuration, you typically also specify --overlap_keyframes to ensure sufficient temporal overlap between windows. The overlap is automatically scaled by the interval, ensuring at least one paired keyframe exists per window boundary.

With window_size = 128, num_scale_frames = 8, and keyframe_interval = 13, a single window effectively covers 8 + 120 × 13 = 1568 actual video frames while maintaining a fixed KV cache size of 128 entries.

Flow-Based Keyframe Mode (Alternative)

Beyond fixed intervals, LingBot-Map supports adaptive keyframe selection via optical flow analysis. Supply a positive flow_threshold to enable this mode, which supersedes the fixed interval setting.

In flow-based mode (implemented in gct_stream_window_v2.py at lines 37-41), frames automatically become keyframes when:

• The optical flow magnitude exceeds flow_threshold, or
• The gap since the last keyframe reaches max_non_keyframe_gap

This dynamic approach preserves detail during rapid motion while compressing static segments.

Why Keyframe Interval Matters for Long Videos

KV cache memory management directly impacts inference capabilities in three critical ways:

1. Memory Scaling: The cache grows linearly with the number of stored keyframes. An interval of 10 reduces memory usage by approximately 90% compared to the default.
2. Pose Quality Boundaries: The RoPE embeddings are trained on sequences up to ~320 frames. Beyond this length without keyframe subsampling, positional encoding degrades and pose estimation accuracy drops.
3. Temporal Range: Windowed inference with aggressive keyframe intervals (e.g., 13-20) allows processing hour-long videos by treating the KV cache as a sliding window over sparse keyframes rather than dense frame history.

Summary

• Default risk: keyframe_interval = 1 stores every frame, exhausting GPU memory on long sequences and exceeding the ~320-frame RoPE training limit.
• Core solution: Set keyframe_interval > 1 to retain only every N-th frame after the initial scale frames, reducing memory usage by factor N.
• Implementation: Configure via --keyframe_interval CLI argument in benchmark_gct_memory.py or the keyframe_interval parameter in inference_streaming().
• Algorithm: Keyframe status is determined by (i - scale_frames) % keyframe_interval == 0 in gct_stream_window_v2.py.
• Advanced option: Use flow_threshold for adaptive keyframe selection based on motion magnitude.
• Windowed mode: Combine with --overlap_keyframes to process effectively unlimited sequence lengths while maintaining cache bounds.

Frequently Asked Questions

What happens to non-keyframe predictions if their KV entries are discarded?

Non-keyframe frames still generate full pose predictions during the forward pass. The model computes attention normally but calls _set_skip_append(True) to prevent writing those specific key/value tensors to the persistent cache. The prediction quality remains high for individual frames, though temporal consistency depends on the density of retained keyframes.

How do I choose the optimal keyframe interval for my video?

Select an interval based on your GPU memory capacity and motion characteristics. For static scenes with slow movement, values between 10-20 work well. For high-motion content, use 3-5 or switch to flow-based mode with flow_threshold=0.5 and max_non_keyframe_gap=30. Always ensure your interval keeps the effective cache size under the ~320-frame RoPE limit mentioned in the README.

Can I use different keyframe intervals for different video segments?

The current implementation in GCTStreamWindowV2 uses a global keyframe_interval parameter for the entire sequence. To achieve variable intervals, you would need to manually implement frame selection logic before calling inference_streaming() with custom keyframe masks, or process the video in chunks with different interval settings and merge results.

Does increasing the keyframe interval affect inference speed?

Yes, but favorably. With fewer keyframes, the KV cache remains smaller, reducing memory bandwidth pressure during attention computation. However, the primary benefit is enabling longer effective context windows; the per-frame computation cost remains constant since non-keyframes still require a full forward pass to generate predictions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}