Scale Frames vs Keyframes in the KV Cache: Lingvo-Bot's Two-Stream Memory Architecture

Scale frames provide a permanent, never-evicted reference segment for global scale estimation, while keyframes are dynamically selected anchor frames that slide through the KV cache window and get evicted when they fall outside the buffer.

Lingvo-Bot's video processing pipeline relies on a specialized KV cache to store intermediate activations and avoid recomputing attention for every frame. Within the FlashInferKVCacheManager implementation in the lingbot-map repository, the system maintains two distinct categories of cached frames—scale frames and keyframes—each serving different computational purposes and following different persistence rules.

What Are Scale Frames in the KV Cache?

Scale frames are a fixed-size prefix of the video sequence (default 8 frames) that provide a bidirectional attention window for global scale and pose estimation. These frames are stored in the dedicated self.scale_patch_pages buffer within the FlashInferKVCacheManager class.

According to the source code comments in lingbot_map/layers/flashinfer_cache.py, scale pages are "never evicted, maxlen=scale_frames". This means once the initial scale frames are processed, they remain in GPU memory for the entire sequence duration, allowing later frames to reference this stable global coordinate system. The count is determined by the kv_cache_scale_frames parameter passed during model initialization in the GCTStream constructor.

What Are Keyframes in the KV Cache?

Keyframes serve as the regular anchor points for causal sliding-window attention. Unlike scale frames, keyframes are stored in self.live_window_patch_pages and are subject to eviction once they fall outside the kv_cache_sliding_window boundary.

The system determines keyframes at runtime using either:

• Fixed intervals: Controlled by keyframe_interval (default 1, meaning every frame)
• Optical flow: When flow_threshold > 0, the _compute_flow_magnitude function calculates motion between frames, and only frames exceeding the threshold are retained as keyframes

Non-keyframes can still attend to the existing cache but are not stored themselves—the manager sets _skip_append = True to prevent allocation.

Key Differences Between Scale Frames and Keyframes

Understanding the architectural separation helps optimize memory usage and inference latency:

• Persistence: Scale frames stored in scale_patch_pages are never evicted and stay for the full sequence, while keyframes in live_window_patch_pages are evicted via FlashInferKVCacheManager.evict_frames when they exceed the sliding window limit.

• Visibility: Scale frames always appear first in the page order (scale_patch_pages → live_window_patch_pages → special_pages), making them universally accessible. Keyframes are only visible while inside the current window.

• Selection Logic: Scale frames are fixed at model startup (kv_cache_scale_frames=8). Keyframes are selected dynamically during GCTStream.inference_streaming based on keyframe_interval modulo arithmetic ((i - scale_frames) % keyframe_interval == 0) or optical flow magnitude.

• Memory Impact: Scale frames consume a constant memory footprint regardless of sequence length, whereas keyframe memory scales with kv_cache_sliding_window divided by the keyframe interval.

Implementation Details in the Source Code

The two-stream architecture is implemented across two primary files:

FlashInferKVCacheManager (lingbot_map/layers/flashinfer_cache.py)

The append_frame method implements the routing logic: if the current frame count is below self.scale_frames, the page goes to scale_patch_pages; otherwise, it routes to live_window_patch_pages. The eviction logic specifically targets only live_window_patch_pages, preserving scale frames indefinitely.

The get_cache_stats method reports the current split between scale_pages and live_pages, allowing runtime monitoring of the cache composition.

GCTStream Inference (lingbot_map/models/gct_stream.py)

Inside inference_streaming, the keyframe decision logic appears around lines 30-57:

is_keyframe = (keyframe_interval <= 1) or ((i - scale_frames) % keyframe_interval == 0)

When flow_threshold is enabled, the system defers eviction decisions using _set_defer_eviction(True) until _compute_flow_magnitude determines if motion warrants keeping the frame. If rejected, _rollback_last_frame removes the tentative allocation.

Practical Configuration Examples

Example 1: Inspecting Cache Statistics

Monitor the split between permanent scale pages and sliding keyframe pages:

from lingbot_map.models.gct_stream import GCTStream

model = GCTStream(kv_cache_scale_frames=8, kv_cache_sliding_window=64)
_ = model.inference_streaming(
    images,  # torch.Tensor [B, S, 3, H, W]

    keyframe_interval=1,
)
stats = model.get_kv_cache_info()

# Returns: {'scale_pages': 8, 'live_pages': <=64, ...}

Example 2: Sparse Keyframes for Memory Efficiency

Reduce memory usage by caching only every 4th frame:

model = GCTStream(kv_cache_scale_frames=8, kv_cache_sliding_window=64)
_ = model.inference_streaming(
    images,
    keyframe_interval=4,  # Only 1/4 of frames stored

)

# Results in: 8 permanent scale pages + max 16 live pages (64/4)

Example 3: Flow-Based Dynamic Selection

Use optical flow to preserve frames only when motion exceeds 2 pixels:

model = GCTStream()
_ = model.inference_streaming(
    images,
    keyframe_interval=1,      # Ignored when flow_threshold > 0

    flow_threshold=2.0,       # Pixels of motion

    max_non_keyframe_gap=10,  # Force keyframe if gap exceeds 10

)

Summary

• Scale frames provide permanent, bidirectional context for global scale estimation and reside in scale_patch_pages without eviction.
• Keyframes act as sliding-window anchors stored in live_window_patch_pages, evicted when exceeding kv_cache_sliding_window.
• Scale frame count is fixed by kv_cache_scale_frames (default 8) at model construction; keyframe density is controlled by keyframe_interval or flow_threshold during inference.
• The FlashInferKVCacheManager maintains separate page tables for each type, with _set_skip_append and _rollback_last_frame handling dynamic keyframe rejection.
• Memory-conscious applications should increase keyframe_interval or enable flow_threshold to minimize live_pages while preserving the constant 8 scale pages.

Frequently Asked Questions

How do scale frames affect memory usage compared to keyframes?

Scale frames consume a fixed allocation of kv_cache_scale_frames pages (typically 8) that remain constant regardless of video length. Keyframes consume variable space up to kv_cache_sliding_window pages, meaning their memory footprint scales with the window size and inversely with the keyframe interval. For long videos, scale frames represent a negligible fraction of total memory compared to the sliding keyframe buffer.

Can I disable scale frames and rely only on keyframes?

No, the architecture requires at least the scale frame segment to establish the global coordinate reference for bidirectional attention. While you can theoretically set kv_cache_scale_frames to a very low value, the model expects this initial segment for reliable scale estimation. Removing it entirely would break the attention pattern in GCTStream.

What happens when optical flow detection skips too many frames?

When using flow_threshold, the system tracks consecutive non-keyframes and enforces a maximum gap via max_non_keyframe_gap. If optical flow stays below the threshold for more frames than this limit, the code automatically promotes the current frame to keyframe status regardless of flow magnitude, ensuring the cache never loses temporal context for too long.

How does the sliding window eviction interact with scale frames?

It does not. The eviction logic in FlashInferKVCacheManager.evict_frames specifically operates only on live_window_patch_pages. Scale frames in scale_patch_pages are excluded from eviction calculations entirely, ensuring the global reference established by the first 8 frames persists even when processing hours of video.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}