# KV Cache Mechanism in LingBot-Map for Streaming Inference: A Deep Dive

> Explore LingBot-Map's paged KV cache mechanism with FlashInfer for linear attention in streaming inference. Learn how it optimizes past token recomputation and speeds up video processing.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: deep-dive
- Published: 2026-07-03

---

**LingBot-Map uses a paged KV cache with FlashInfer to enable linear-time attention during streaming video inference, avoiding recomputation of past frame tokens through a dual-stream architecture that separates recyclable patch tokens from permanent special tokens.**

LingBot-Map performs online frame-by-frame inference on streaming video using a causal transformer architecture. To achieve efficient streaming without reprocessing the entire video history at each step, the repository implements a sophisticated **KV cache mechanism** that stores key-value pairs from previous frames in a paged memory structure. This implementation leverages FlashInfer CUDA kernels to manage two distinct token streams—patch and special—while supporting sliding-window eviction and flow-based keyframe selection.

## Paged KV Cache Architecture

The core implementation resides in `FlashInferKVCacheManager` within [[`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py)](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py). This manager allocates KV tensors across logical pages using FlashInfer's fast CUDA kernels for paged attention, significantly reducing memory overhead compared to dense attention matrices.

The high-level model (`GCTStream`) instantiated in [[`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py)](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py) creates an `AggregatorStream` (defined in [[`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py)](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py)) that owns the cache manager. During each forward pass, the model attends to cached KV tokens from previous frames plus current frame tokens, achieving **O(n)** time complexity per step rather than **O(n²)**.

## Two-Stream Design: Patch vs. Special Tokens

The cache separates tokens into two logical streams to optimize memory usage and eviction policies.

### Patch Stream (Recyclable)

Contains actual image patch tokens for each frame. The manager maintains:

- **Scale pages**: First `kv_cache_scale_frames` frames (default 8) remain permanently resident
- **Sliding window**: Recent frames kept up to `kv_cache_sliding_window` (default 64)
- **Free list**: Evicted pages recycled for new frames via `append_frame`

### Special Stream (Append-Only)

Stores six special tokens (camera, register, scale) per frame. These pages are never evicted, ensuring critical metadata persists across the entire sequence. Because special tokens consume minimal memory, this append-only approach avoids complex garbage collection while maintaining temporal coherence.

## Sliding-Window Eviction Strategy

After processing each frame via `append_frame`, the system calls `evict_frames` to enforce memory bounds. The eviction logic:

1. Checks if `live_window_patch_pages` exceeds `kv_cache_sliding_window`
2. Pops the oldest window page and returns it to the free list
3. Preserves all scale pages and special stream pages regardless of age

This guarantees that bidirectional scale-token processing always has access to reference frames while limiting the computational context to recent history.

## Flow-Based Keyframe Selection

LingBot-Map supports intelligent keyframe selection through deferred eviction. When optical flow magnitude exceeds a threshold during `inference_streaming`:

1. Eviction is temporarily disabled via `_set_defer_eviction(True)`
2. If the frame qualifies as a keyframe, `execute_deferred_eviction` commits the KV pairs
3. Otherwise, `rollback_last_frame` discards the newly added KV tensors

This mechanism, implemented in `GCTStream.inference_streaming`, allows dynamic temporal sampling without breaking the causal attention pattern.

## Configuration Parameters

The following parameters in `GCTStream.__init__` control cache behavior:

- **`kv_cache_sliding_window`**: Number of recent frames retained in the patch window (default: 64)
- **`kv_cache_scale_frames`**: Permanent resident scale frames (default: 8)
- **`kv_cache_cross_frame_special`**: Whether to retain special tokens from evicted frames for cross-frame attention (default: True)
- **`kv_cache_include_scale_frames`**: Include scale frames in cache (default: True)
- **`kv_cache_camera_only`**: Restrict cache to camera tokens only, excluding patches (default: False)

## Practical Implementation Examples

### Initializing a Model with Custom Cache Settings

```python
import torch
from lingbot_map.models.gct_stream_window_v2 import GCTStream

model = GCTStream(
    img_size=518,
    patch_size=14,
    embed_dim=1024,
    kv_cache_sliding_window=128,  # Larger context window

    kv_cache_scale_frames=4,     # Fewer permanent scale frames

    kv_cache_cross_frame_special=False,
    enable_3d_rope=True,
)
model.eval()

```

### Streaming Inference with Automatic Cache Management

```python

# Process 120 frames with keyframe every 5 frames

imgs = torch.randn(1, 120, 3, 518, 518)

preds = model.inference_streaming(
    images=imgs,
    keyframe_interval=5,
    output_device=torch.device('cpu'),
)

```

The model automatically handles cache initialization via `clean_kv_cache`, frame appending, and eviction according to the configured sliding window.

### Manual Cache Control

```python

# Reset cache when switching videos

model.clean_kv_cache()

# Process frame without storing KV (non-keyframe)

model._set_skip_append(True)
frame_pred = model.forward(frame_tensor)
model._set_skip_append(False)

```

### Inspecting Cache Statistics

```python
stats = model.aggregator.kv_cache_manager.get_cache_stats(block_idx=0)
print(f"Live pages: {stats['live_pages']}, Scale pages: {stats['scale_pages']}")

```

## Summary

- **LingBot-Map** implements a **paged KV cache** using FlashInfer in [`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py) to enable efficient streaming video inference.
- The **two-stream architecture** separates recyclable image patches from permanent special tokens, optimizing memory usage while preserving critical metadata.
- **Sliding-window eviction** maintains linear time complexity by recycling old patch pages while keeping scale frames permanently resident.
- **Flow-based keyframe selection** uses deferred eviction (`execute_deferred_eviction` and `rollback_last_frame`) to dynamically select important frames based on motion.
- Configuration parameters like `kv_cache_sliding_window` and `kv_cache_scale_frames` allow fine-tuning of the memory-accuracy tradeoff.

## Frequently Asked Questions

### How does the KV cache in LingBot-Map differ from standard transformer KV caching?

Standard implementations cache all previous tokens indefinitely, leading to quadratic memory growth. LingBot-Map uses a **paged cache with sliding-window eviction** that maintains constant memory usage by recycling old patch pages while preserving essential scale and special tokens. This design specifically supports unbounded video streams without memory exhaustion.

### What is the purpose of the two-stream design in the KV cache?

The **patch stream** handles recyclable image tokens subject to sliding-window eviction, while the **special stream** maintains permanent tokens for camera parameters and registers. This separation ensures that critical metadata persists across all frames while allowing aggressive memory management for high-dimensional visual features.

### Can I disable the sliding window to process very long videos?

While you can increase `kv_cache_sliding_window` arbitrarily, disabling eviction entirely would cause out-of-memory errors on long videos. The scale frames (`kv_cache_scale_frames`) provide a compromise by maintaining bidirectional context for key reference frames while the sliding window handles recent temporal context. For truly unbounded streams, the current eviction mechanism is required.

### How does flow-based keyframe selection interact with the cache?

When optical flow exceeds a threshold, the model temporarily suspends eviction via `_set_defer_eviction(True)`. If the frame is selected as a keyframe, the KV tensors are committed; otherwise, `rollback_last_frame` removes them. This allows the cache to dynamically retain high-motion frames while discarding redundant static content, optimizing both memory and computational resources.