Keyframes and Non-keyframes in LingBot-Map's KV Cache: Memory-Efficient Streaming

Keyframes in LingBot-Map are frames whose attention keys and values (KV) are permanently retained in the cache for future attention operations, while non-keyframes are frames whose KV tensors are computed for the current forward pass but discarded immediately afterward to limit memory growth.

The Robbyant/lingbot-map repository implements a streaming video understanding architecture that processes long visual sequences without recomputing attention for every historical frame. At the heart of this efficiency lies a selective KV cache mechanism that categorizes incoming frames as either keyframes or non-keyframes, enabling scalable online inference with controlled memory footprint.

What the KV Cache Stores

During streaming inference, each transformer block generates attention keys (K) and values (V) for every input frame. In lingbot_map/models/gct_stream.py, these tensors are appended to a running cache that persists across forward passes. When processing subsequent frames, the model performs attention over the concatenated cache [cached_K, K_current] and [cached_V, V_current], allowing the model to reference past temporal context without redundant computation.

The cache itself resides in AggregatorStream.kv_cache (defined in lingbot_map/aggregator/stream.py) as a dictionary structure, or alternatively within a FlashInfer cache manager (lingbot_map/layers/flashinfer_cache.py). Both implementations support conditional storage logic that distinguishes between frame types.

Keyframes vs Non-keyframes: Core Differences

The fundamental distinction determines whether computed KV tensors survive beyond the current inference step:

Frame Type	Cache Behavior	KV Handling
Keyframe	KV tensors remain in cache after forward pass	_skip_append flag remains False; tensors are appended permanently
Non-keyframe	KV tensors are discarded after use	_skip_append is set to True temporarily via _set_skip_append(True)

During the forward pass, the model still computes full predictions for non-keyframes and attends to their KV tensors for the current frame. However, by setting the internal _skip_append flag to True before the cache update operation, the system prevents these tensors from being added to the persistent cache, effectively reducing memory consumption.

Keyframe Selection Strategies

LingBot-Map supports two primary modes for determining which frames become keyframes, implemented in lingbot_map/models/gct_stream_window_v2.py.

Fixed-Interval Keyframe Mode

Controlled by the keyframe_interval parameter, this mode treats every N-th frame as a keyframe after the initial scale-frame phase. According to the source code in lines 30-36, setting keyframe_interval=4 means only 1 in 4 frames occupies cache space, reducing memory growth by approximately 75%.

Flow-Based Keyframe Mode

Activated when flow_threshold > 0 (lines 37-41), this dynamic mode computes the mean optical-flow magnitude between the current frame and the last keyframe. If the motion exceeds the threshold—or if the gap between keyframes reaches max_non_keyframe_gap—the current frame is promoted to keyframe status. This adaptive approach ensures that significant motion events are preserved in the cache while static segments use minimal memory.

Memory Efficiency and Implementation

Non-keyframes exist primarily to mitigate the quadratic memory growth of full attention caches over long video sequences. By caching only a subset of frames (the keyframes), the model maintains constant memory usage proportional to total_frames / keyframe_interval rather than total_frames.

The internal _set_skip_append(skip: bool) method (lines 404-409 in gct_stream_window_v2.py) coordinates this behavior across all KV cache instances, including both the aggregator and the camera head (lingbot_map/heads/camera_head.py). When invoked with True, the method signals the cache implementations to compute attention normally but skip the append operation, effectively creating a non-keyframe.

Practical Usage Examples

Fixed-Interval Streaming

Process video with a keyframe every 5 frames to limit cache growth:

preds = model.inference_streaming(
    images,               # Tensor [B, S, 3, H, W]

    keyframe_interval=5,  # Keep KV only on every 5th frame

    flow_threshold=0.0,   # Disable flow-based mode

)

Flow-Based Keyframe Selection

Enable motion-aware keyframing to capture significant visual changes:

preds = model.inference_streaming(
    images,
    flow_threshold=2.0,        # Enable flow-based mode

    max_non_keyframe_gap=15,   # Force keyframe every 15 frames maximum

)

Manual Non-Keyframe Control

For custom inference loops, directly manipulate the cache storage flag:

model._set_skip_append(True)          # Prevent KV storage for next frame

output = model.forward(frame_tensor)  # Process frame

model._set_skip_append(False)         # Reset for subsequent frames

Summary

• Keyframes retain their KV tensors in the cache permanently, providing stable long-range temporal context for future attention operations.
• Non-keyframes compute KV for immediate attention but discard them afterward, reducing memory consumption by a factor equal to the keyframe_interval.
• The _set_skip_append() method in gct_stream_window_v2.py controls this behavior internally, while inference_streaming() exposes fixed-interval and flow-based selection strategies.
• Core implementations reside in lingbot_map/aggregator/stream.py (cache storage), lingbot_map/layers/flashinfer_cache.py (optimized kernels), and lingbot_map/models/gct_stream_window_v2.py (selection logic).

Frequently Asked Questions

What determines if a frame becomes a keyframe in LingBot-Map?

A frame becomes a keyframe based on either fixed intervals or optical flow magnitude. In fixed-interval mode (controlled by keyframe_interval), every N-th frame is a keyframe. In flow-based mode (activated by flow_threshold > 0), a frame becomes a keyframe if the mean optical flow from the previous keyframe exceeds the threshold or if the maximum gap of non-keyframes is reached.

How does the _set_skip_append method control the KV cache?

The _set_skip_append(skip: bool) method toggles an internal flag that instructs the KV cache implementations (in both AggregatorStream and the camera head) whether to store the current frame's KV tensors. When set to True for non-keyframes, the caches compute attention using the current KV but do not append them to persistent storage, whereas False permits normal caching for keyframes.

Can I mix fixed-interval and flow-based keyframe selection?

While the implementation in inference_streaming() accepts both keyframe_interval and flow_threshold parameters, they typically operate as mutually exclusive strategies. Setting flow_threshold > 0 enables the flow-based mode, which generally overrides fixed-interval logic to prioritize motion-aware keyframe selection. For hybrid behavior, you would need to implement custom logic using the lower-level _set_skip_append() API.

Which files contain the core KV cache logic for streaming inference?

The primary selection logic resides in lingbot_map/models/gct_stream_window_v2.py, with the base streaming implementation in lingbot_map/models/gct_stream.py. The cache storage mechanisms are defined in lingbot_map/aggregator/stream.py (dictionary-based cache) and lingbot_map/layers/flashinfer_cache.py (FlashInfer optimized caches). The camera head's parallel cache is managed in lingbot_map/heads/camera_head.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}