# Keyframes and Non-keyframes in LingBot-Map's KV Cache: Memory-Efficient Streaming

> Discover keyframes and non-keyframes in LingBot-Map's KV cache. Learn how these techniques optimize memory for efficient streaming in the Robbyant/lingbot-map repository.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: internals
- Published: 2026-07-03

---

**Keyframes in LingBot-Map are frames whose attention keys and values (KV) are permanently retained in the cache for future attention operations, while non-keyframes are frames whose KV tensors are computed for the current forward pass but discarded immediately afterward to limit memory growth.**

The `Robbyant/lingbot-map` repository implements a streaming video understanding architecture that processes long visual sequences without recomputing attention for every historical frame. At the heart of this efficiency lies a selective KV cache mechanism that categorizes incoming frames as either keyframes or non-keyframes, enabling scalable online inference with controlled memory footprint.

## What the KV Cache Stores

During streaming inference, each transformer block generates attention keys (`K`) and values (`V`) for every input frame. In [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py), these tensors are appended to a running cache that persists across forward passes. When processing subsequent frames, the model performs attention over the concatenated cache `[cached_K, K_current]` and `[cached_V, V_current]`, allowing the model to reference past temporal context without redundant computation.

The cache itself resides in `AggregatorStream.kv_cache` (defined in [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py)) as a dictionary structure, or alternatively within a FlashInfer cache manager ([`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py)). Both implementations support conditional storage logic that distinguishes between frame types.

## Keyframes vs Non-keyframes: Core Differences

The fundamental distinction determines whether computed KV tensors survive beyond the current inference step:

| Frame Type | Cache Behavior | KV Handling |
|------------|---------------|-------------|
| **Keyframe** | KV tensors **remain** in cache after forward pass | `_skip_append` flag remains `False`; tensors are appended permanently |
| **Non-keyframe** | KV tensors are **discarded** after use | `_skip_append` is set to `True` temporarily via `_set_skip_append(True)` |

During the forward pass, the model still computes full predictions for non-keyframes and attends to their KV tensors for the current frame. However, by setting the internal `_skip_append` flag to `True` before the cache update operation, the system prevents these tensors from being added to the persistent cache, effectively reducing memory consumption.

## Keyframe Selection Strategies

LingBot-Map supports two primary modes for determining which frames become keyframes, implemented in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py).

### Fixed-Interval Keyframe Mode

Controlled by the `keyframe_interval` parameter, this mode treats every *N*-th frame as a keyframe after the initial scale-frame phase. According to the source code in lines 30-36, setting `keyframe_interval=4` means only 1 in 4 frames occupies cache space, reducing memory growth by approximately 75%.

### Flow-Based Keyframe Mode

Activated when `flow_threshold > 0` (lines 37-41), this dynamic mode computes the mean optical-flow magnitude between the current frame and the last keyframe. If the motion exceeds the threshold—or if the gap between keyframes reaches `max_non_keyframe_gap`—the current frame is promoted to keyframe status. This adaptive approach ensures that significant motion events are preserved in the cache while static segments use minimal memory.

## Memory Efficiency and Implementation

Non-keyframes exist primarily to mitigate the quadratic memory growth of full attention caches over long video sequences. By caching only a subset of frames (the keyframes), the model maintains constant memory usage proportional to `total_frames / keyframe_interval` rather than `total_frames`.

The internal `_set_skip_append(skip: bool)` method (lines 404-409 in [`gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream_window_v2.py)) coordinates this behavior across all KV cache instances, including both the aggregator and the camera head ([`lingbot_map/heads/camera_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/camera_head.py)). When invoked with `True`, the method signals the cache implementations to compute attention normally but skip the append operation, effectively creating a non-keyframe.

## Practical Usage Examples

### Fixed-Interval Streaming

Process video with a keyframe every 5 frames to limit cache growth:

```python
preds = model.inference_streaming(
    images,               # Tensor [B, S, 3, H, W]

    keyframe_interval=5,  # Keep KV only on every 5th frame

    flow_threshold=0.0,   # Disable flow-based mode

)

```

### Flow-Based Keyframe Selection

Enable motion-aware keyframing to capture significant visual changes:

```python
preds = model.inference_streaming(
    images,
    flow_threshold=2.0,        # Enable flow-based mode

    max_non_keyframe_gap=15,   # Force keyframe every 15 frames maximum

)

```

### Manual Non-Keyframe Control

For custom inference loops, directly manipulate the cache storage flag:

```python
model._set_skip_append(True)          # Prevent KV storage for next frame

output = model.forward(frame_tensor)  # Process frame

model._set_skip_append(False)         # Reset for subsequent frames

```

## Summary

- **Keyframes** retain their KV tensors in the cache permanently, providing stable long-range temporal context for future attention operations.
- **Non-keyframes** compute KV for immediate attention but discard them afterward, reducing memory consumption by a factor equal to the `keyframe_interval`.
- The `_set_skip_append()` method in [`gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream_window_v2.py) controls this behavior internally, while `inference_streaming()` exposes fixed-interval and flow-based selection strategies.
- Core implementations reside in [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py) (cache storage), [`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py) (optimized kernels), and [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py) (selection logic).

## Frequently Asked Questions

### What determines if a frame becomes a keyframe in LingBot-Map?

A frame becomes a keyframe based on either fixed intervals or optical flow magnitude. In fixed-interval mode (controlled by `keyframe_interval`), every *N*-th frame is a keyframe. In flow-based mode (activated by `flow_threshold > 0`), a frame becomes a keyframe if the mean optical flow from the previous keyframe exceeds the threshold or if the maximum gap of non-keyframes is reached.

### How does the `_set_skip_append` method control the KV cache?

The `_set_skip_append(skip: bool)` method toggles an internal flag that instructs the KV cache implementations (in both `AggregatorStream` and the camera head) whether to store the current frame's KV tensors. When set to `True` for non-keyframes, the caches compute attention using the current KV but do not append them to persistent storage, whereas `False` permits normal caching for keyframes.

### Can I mix fixed-interval and flow-based keyframe selection?

While the implementation in `inference_streaming()` accepts both `keyframe_interval` and `flow_threshold` parameters, they typically operate as mutually exclusive strategies. Setting `flow_threshold > 0` enables the flow-based mode, which generally overrides fixed-interval logic to prioritize motion-aware keyframe selection. For hybrid behavior, you would need to implement custom logic using the lower-level `_set_skip_append()` API.

### Which files contain the core KV cache logic for streaming inference?

The primary selection logic resides in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py), with the base streaming implementation in [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py). The cache storage mechanisms are defined in [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py) (dictionary-based cache) and [`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py) (FlashInfer optimized caches). The camera head's parallel cache is managed in [`lingbot_map/heads/camera_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/camera_head.py).