KV Cache Mechanism in LingBot-Map for Streaming Inference: A Deep Dive

LingBot-Map uses a paged KV cache with FlashInfer to enable linear-time attention during streaming video inference, avoiding recomputation of past frame tokens through a dual-stream architecture that separates recyclable patch tokens from permanent special tokens.

LingBot-Map performs online frame-by-frame inference on streaming video using a causal transformer architecture. To achieve efficient streaming without reprocessing the entire video history at each step, the repository implements a sophisticated KV cache mechanism that stores key-value pairs from previous frames in a paged memory structure. This implementation leverages FlashInfer CUDA kernels to manage two distinct token streams—patch and special—while supporting sliding-window eviction and flow-based keyframe selection.

Paged KV Cache Architecture

The core implementation resides in FlashInferKVCacheManager within [lingbot_map/layers/flashinfer_cache.py](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py). This manager allocates KV tensors across logical pages using FlashInfer's fast CUDA kernels for paged attention, significantly reducing memory overhead compared to dense attention matrices.

The high-level model (GCTStream) instantiated in [lingbot_map/models/gct_stream_window_v2.py](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py) creates an AggregatorStream (defined in [lingbot_map/aggregator/stream.py](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py)) that owns the cache manager. During each forward pass, the model attends to cached KV tokens from previous frames plus current frame tokens, achieving O(n) time complexity per step rather than O(n²).

Two-Stream Design: Patch vs. Special Tokens

The cache separates tokens into two logical streams to optimize memory usage and eviction policies.

Patch Stream (Recyclable)

Contains actual image patch tokens for each frame. The manager maintains:

• Scale pages: First kv_cache_scale_frames frames (default 8) remain permanently resident
• Sliding window: Recent frames kept up to kv_cache_sliding_window (default 64)
• Free list: Evicted pages recycled for new frames via append_frame

Special Stream (Append-Only)

Stores six special tokens (camera, register, scale) per frame. These pages are never evicted, ensuring critical metadata persists across the entire sequence. Because special tokens consume minimal memory, this append-only approach avoids complex garbage collection while maintaining temporal coherence.

Sliding-Window Eviction Strategy

After processing each frame via append_frame, the system calls evict_frames to enforce memory bounds. The eviction logic:

1. Checks if live_window_patch_pages exceeds kv_cache_sliding_window
2. Pops the oldest window page and returns it to the free list
3. Preserves all scale pages and special stream pages regardless of age

This guarantees that bidirectional scale-token processing always has access to reference frames while limiting the computational context to recent history.

Flow-Based Keyframe Selection

LingBot-Map supports intelligent keyframe selection through deferred eviction. When optical flow magnitude exceeds a threshold during inference_streaming:

1. Eviction is temporarily disabled via _set_defer_eviction(True)
2. If the frame qualifies as a keyframe, execute_deferred_eviction commits the KV pairs
3. Otherwise, rollback_last_frame discards the newly added KV tensors

This mechanism, implemented in GCTStream.inference_streaming, allows dynamic temporal sampling without breaking the causal attention pattern.

Configuration Parameters

The following parameters in GCTStream.__init__ control cache behavior:

• kv_cache_sliding_window: Number of recent frames retained in the patch window (default: 64)
• kv_cache_scale_frames: Permanent resident scale frames (default: 8)
• kv_cache_cross_frame_special: Whether to retain special tokens from evicted frames for cross-frame attention (default: True)
• kv_cache_include_scale_frames: Include scale frames in cache (default: True)
• kv_cache_camera_only: Restrict cache to camera tokens only, excluding patches (default: False)

Practical Implementation Examples

Initializing a Model with Custom Cache Settings

import torch
from lingbot_map.models.gct_stream_window_v2 import GCTStream

model = GCTStream(
    img_size=518,
    patch_size=14,
    embed_dim=1024,
    kv_cache_sliding_window=128,  # Larger context window

    kv_cache_scale_frames=4,     # Fewer permanent scale frames

    kv_cache_cross_frame_special=False,
    enable_3d_rope=True,
)
model.eval()

Streaming Inference with Automatic Cache Management

# Process 120 frames with keyframe every 5 frames

imgs = torch.randn(1, 120, 3, 518, 518)

preds = model.inference_streaming(
    images=imgs,
    keyframe_interval=5,
    output_device=torch.device('cpu'),
)

The model automatically handles cache initialization via clean_kv_cache, frame appending, and eviction according to the configured sliding window.

Manual Cache Control

# Reset cache when switching videos

model.clean_kv_cache()

# Process frame without storing KV (non-keyframe)

model._set_skip_append(True)
frame_pred = model.forward(frame_tensor)
model._set_skip_append(False)

Inspecting Cache Statistics

stats = model.aggregator.kv_cache_manager.get_cache_stats(block_idx=0)
print(f"Live pages: {stats['live_pages']}, Scale pages: {stats['scale_pages']}")

Summary

• LingBot-Map implements a paged KV cache using FlashInfer in lingbot_map/layers/flashinfer_cache.py to enable efficient streaming video inference.
• The two-stream architecture separates recyclable image patches from permanent special tokens, optimizing memory usage while preserving critical metadata.
• Sliding-window eviction maintains linear time complexity by recycling old patch pages while keeping scale frames permanently resident.
• Flow-based keyframe selection uses deferred eviction (execute_deferred_eviction and rollback_last_frame) to dynamically select important frames based on motion.
• Configuration parameters like kv_cache_sliding_window and kv_cache_scale_frames allow fine-tuning of the memory-accuracy tradeoff.

Frequently Asked Questions

How does the KV cache in LingBot-Map differ from standard transformer KV caching?

Standard implementations cache all previous tokens indefinitely, leading to quadratic memory growth. LingBot-Map uses a paged cache with sliding-window eviction that maintains constant memory usage by recycling old patch pages while preserving essential scale and special tokens. This design specifically supports unbounded video streams without memory exhaustion.

What is the purpose of the two-stream design in the KV cache?

The patch stream handles recyclable image tokens subject to sliding-window eviction, while the special stream maintains permanent tokens for camera parameters and registers. This separation ensures that critical metadata persists across all frames while allowing aggressive memory management for high-dimensional visual features.

Can I disable the sliding window to process very long videos?

While you can increase kv_cache_sliding_window arbitrarily, disabling eviction entirely would cause out-of-memory errors on long videos. The scale frames (kv_cache_scale_frames) provide a compromise by maintaining bidirectional context for key reference frames while the sliding window handles recent temporal context. For truly unbounded streams, the current eviction mechanism is required.

How does flow-based keyframe selection interact with the cache?

When optical flow exceeds a threshold, the model temporarily suspends eviction via _set_defer_eviction(True). If the frame is selected as a keyframe, the KV tensors are committed; otherwise, rollback_last_frame removes them. This allows the cache to dynamically retain high-motion frames while discarding redundant static content, optimizing both memory and computational resources.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}