# Scale Frames vs Keyframes in the KV Cache: Lingvo-Bot's Two-Stream Memory Architecture

> Understand the difference between scale frames and keyframes in the KV cache. Lingvo-Bot utilizes scale frames for global estimation and dynamic keyframes for memory efficiency.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: deep-dive
- Published: 2026-07-04

---

**Scale frames provide a permanent, never-evicted reference segment for global scale estimation, while keyframes are dynamically selected anchor frames that slide through the KV cache window and get evicted when they fall outside the buffer.**

Lingvo-Bot's video processing pipeline relies on a specialized **KV cache** to store intermediate activations and avoid recomputing attention for every frame. Within the `FlashInferKVCacheManager` implementation in the `lingbot-map` repository, the system maintains two distinct categories of cached frames—**scale frames** and **keyframes**—each serving different computational purposes and following different persistence rules.

## What Are Scale Frames in the KV Cache?

**Scale frames** are a fixed-size prefix of the video sequence (default **8 frames**) that provide a bidirectional attention window for global scale and pose estimation. These frames are stored in the dedicated `self.scale_patch_pages` buffer within the `FlashInferKVCacheManager` class.

According to the source code comments in [`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py), scale pages are "*never evicted, maxlen=scale_frames*". This means once the initial scale frames are processed, they remain in GPU memory for the entire sequence duration, allowing later frames to reference this stable global coordinate system. The count is determined by the `kv_cache_scale_frames` parameter passed during model initialization in the `GCTStream` constructor.

## What Are Keyframes in the KV Cache?

**Keyframes** serve as the regular anchor points for causal sliding-window attention. Unlike scale frames, keyframes are stored in `self.live_window_patch_pages` and are subject to eviction once they fall outside the `kv_cache_sliding_window` boundary.

The system determines keyframes at runtime using either:
- **Fixed intervals**: Controlled by `keyframe_interval` (default 1, meaning every frame)
- **Optical flow**: When `flow_threshold > 0`, the `_compute_flow_magnitude` function calculates motion between frames, and only frames exceeding the threshold are retained as keyframes

Non-keyframes can still attend to the existing cache but are not stored themselves—the manager sets `_skip_append = True` to prevent allocation.

## Key Differences Between Scale Frames and Keyframes

Understanding the architectural separation helps optimize memory usage and inference latency:

- **Persistence**: Scale frames stored in `scale_patch_pages` are never evicted and stay for the full sequence, while keyframes in `live_window_patch_pages` are evicted via `FlashInferKVCacheManager.evict_frames` when they exceed the sliding window limit.

- **Visibility**: Scale frames always appear first in the page order (`scale_patch_pages` → `live_window_patch_pages` → `special_pages`), making them universally accessible. Keyframes are only visible while inside the current window.

- **Selection Logic**: Scale frames are fixed at model startup (`kv_cache_scale_frames=8`). Keyframes are selected dynamically during `GCTStream.inference_streaming` based on `keyframe_interval` modulo arithmetic (`(i - scale_frames) % keyframe_interval == 0`) or optical flow magnitude.

- **Memory Impact**: Scale frames consume a constant memory footprint regardless of sequence length, whereas keyframe memory scales with `kv_cache_sliding_window` divided by the keyframe interval.

## Implementation Details in the Source Code

The two-stream architecture is implemented across two primary files:

### FlashInferKVCacheManager ([`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py))

The `append_frame` method implements the routing logic: if the current frame count is below `self.scale_frames`, the page goes to `scale_patch_pages`; otherwise, it routes to `live_window_patch_pages`. The eviction logic specifically targets only `live_window_patch_pages`, preserving scale frames indefinitely.

The `get_cache_stats` method reports the current split between `scale_pages` and `live_pages`, allowing runtime monitoring of the cache composition.

### GCTStream Inference ([`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py))

Inside `inference_streaming`, the keyframe decision logic appears around lines 30-57:

```python
is_keyframe = (keyframe_interval <= 1) or ((i - scale_frames) % keyframe_interval == 0)

```

When `flow_threshold` is enabled, the system defers eviction decisions using `_set_defer_eviction(True)` until `_compute_flow_magnitude` determines if motion warrants keeping the frame. If rejected, `_rollback_last_frame` removes the tentative allocation.

## Practical Configuration Examples

### Example 1: Inspecting Cache Statistics

Monitor the split between permanent scale pages and sliding keyframe pages:

```python
from lingbot_map.models.gct_stream import GCTStream

model = GCTStream(kv_cache_scale_frames=8, kv_cache_sliding_window=64)
_ = model.inference_streaming(
    images,  # torch.Tensor [B, S, 3, H, W]

    keyframe_interval=1,
)
stats = model.get_kv_cache_info()

# Returns: {'scale_pages': 8, 'live_pages': <=64, ...}

```

### Example 2: Sparse Keyframes for Memory Efficiency

Reduce memory usage by caching only every 4th frame:

```python
model = GCTStream(kv_cache_scale_frames=8, kv_cache_sliding_window=64)
_ = model.inference_streaming(
    images,
    keyframe_interval=4,  # Only 1/4 of frames stored

)

# Results in: 8 permanent scale pages + max 16 live pages (64/4)

```

### Example 3: Flow-Based Dynamic Selection

Use optical flow to preserve frames only when motion exceeds 2 pixels:

```python
model = GCTStream()
_ = model.inference_streaming(
    images,
    keyframe_interval=1,      # Ignored when flow_threshold > 0

    flow_threshold=2.0,       # Pixels of motion

    max_non_keyframe_gap=10,  # Force keyframe if gap exceeds 10

)

```

## Summary

- **Scale frames** provide permanent, bidirectional context for global scale estimation and reside in `scale_patch_pages` without eviction.
- **Keyframes** act as sliding-window anchors stored in `live_window_patch_pages`, evicted when exceeding `kv_cache_sliding_window`.
- Scale frame count is fixed by `kv_cache_scale_frames` (default 8) at model construction; keyframe density is controlled by `keyframe_interval` or `flow_threshold` during inference.
- The `FlashInferKVCacheManager` maintains separate page tables for each type, with `_set_skip_append` and `_rollback_last_frame` handling dynamic keyframe rejection.
- Memory-conscious applications should increase `keyframe_interval` or enable `flow_threshold` to minimize `live_pages` while preserving the constant 8 scale pages.

## Frequently Asked Questions

### How do scale frames affect memory usage compared to keyframes?

Scale frames consume a fixed allocation of `kv_cache_scale_frames` pages (typically 8) that remain constant regardless of video length. Keyframes consume variable space up to `kv_cache_sliding_window` pages, meaning their memory footprint scales with the window size and inversely with the keyframe interval. For long videos, scale frames represent a negligible fraction of total memory compared to the sliding keyframe buffer.

### Can I disable scale frames and rely only on keyframes?

No, the architecture requires at least the scale frame segment to establish the global coordinate reference for bidirectional attention. While you can theoretically set `kv_cache_scale_frames` to a very low value, the model expects this initial segment for reliable scale estimation. Removing it entirely would break the attention pattern in `GCTStream`.

### What happens when optical flow detection skips too many frames?

When using `flow_threshold`, the system tracks consecutive non-keyframes and enforces a maximum gap via `max_non_keyframe_gap`. If optical flow stays below the threshold for more frames than this limit, the code automatically promotes the current frame to keyframe status regardless of flow magnitude, ensuring the cache never loses temporal context for too long.

### How does the sliding window eviction interact with scale frames?

It does not. The eviction logic in `FlashInferKVCacheManager.evict_frames` specifically operates only on `live_window_patch_pages`. Scale frames in `scale_patch_pages` are excluded from eviction calculations entirely, ensuring the global reference established by the first 8 frames persists even when processing hours of video.