# How the KV Cache Enables Efficient Streaming Inference in LingBot-Map

> Discover how LingBot-Map's paged KV cache and FlashInfer optimize streaming inference by eliminating redundant attention calculations for linear-time frame processing.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: internals
- Published: 2026-07-06

---

**LingBot-Map uses a two-stream paged KV cache built on FlashInfer to store key-value pairs from previously processed video frames, eliminating redundant attention computations and enabling linear-time scaling for frame-by-frame inference.**

The LingBot-Map repository implements a causal transformer architecture for online video understanding. To achieve real-time performance during streaming inference, the model leverages a sophisticated key-value (KV) cache mechanism that avoids recomputing attention over entire video histories. This article examines how the paged KV cache architecture in [`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py) enables efficient, stateful processing of long video sequences.

## The Two-Stream Paged KV Cache Architecture

At the core of LingBot-Map’s streaming capability is the `FlashInferKVCacheManager` class, which implements a dual-stream memory management system. According to the source code in [`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py), the cache splits tokens into two distinct logical streams to optimize memory usage and attention computation.

### Patch Stream and Special Stream Management

The cache maintains separate handling for visual content and metadata tokens:

- **Patch Stream (Recyclable):** Contains the actual image patch embeddings for each frame. This stream uses a sliding window approach where the first `kv_cache_scale_frames` pages remain permanently resident in memory, while subsequent pages follow a first-in-first-out eviction policy based on the configured window size.

- **Special Stream (Append-Only):** Stores the six special tokens (camera, register, and scale tokens) associated with each frame. These pages are never evicted and accumulate throughout the video sequence. Because special tokens represent minimal memory overhead compared to image patches, this append-only design preserves critical metadata without significant cost.

During each forward pass, the manager constructs a **visible-page table** combining scale pages, live window pages, and all special pages. FlashInfer then executes a single batched attention operation over this concatenated view, as implemented in the `compute_attention` method.

### FlashInfer Integration

The cache manager delegates high-performance attention computation to **FlashInfer**, a CUDA kernel library optimized for paged attention. The `AggregatorStream` class in [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py) instantiates the cache manager and coordinates between the transformer blocks and the FlashInfer backend. This integration allows LingBot-Map to perform memory-efficient attention without materializing the full sequence length in contiguous memory.

## Sliding-Window Eviction for Linear-Time Scaling

To prevent memory from growing unbounded with video length, LingBot-Map implements aggressive eviction through the `evict_frames` method in [`flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/flashinfer_cache.py).

After each frame’s key-value tensors are appended via `append_frame`, the system checks whether the live window exceeds `kv_cache_sliding_window`. When the limit is reached, the oldest window page is removed from `live_window_patch_pages` and returned to a free list for reuse. This mechanism ensures that memory consumption remains constant regardless of video duration, yielding **O(1)** memory usage and **O(n)** computation time relative to sequence length.

**Scale pages remain permanently exempt from eviction.** These resident pages, configured via `kv_cache_scale_frames`, provide stable bidirectional context necessary for processing scale tokens across the entire sequence.

## Flow-Based Keyframe Selection with Deferred Eviction

LingBot-Map supports intelligent keyframe selection through optical flow analysis, implemented via deferred eviction mechanisms in the cache manager.

When `inference_streaming` operates with a flow threshold, the model temporarily disables eviction by calling `_set_defer_eviction(True)` before processing each candidate frame. If the computed optical flow magnitude exceeds the threshold, the frame qualifies as a keyframe, and the system executes `_execute_deferred_eviction` to commit the KV pairs permanently. If the flow magnitude falls below the threshold, the model invokes `_rollback_last_frame` to discard the provisional KV tensors, effectively skipping the non-informative frame without polluting the cache.

This flow-based filtering prevents redundant computation on static or low-motion segments while preserving semantic context for significant scene changes.

## Configuring the KV Cache Parameters

The `GCTStream` class in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py) exposes several configuration knobs for tuning cache behavior:

| Parameter | Description | Default |
|-----------|-------------|---------|
| `kv_cache_sliding_window` | Number of recent patch frames retained in the sliding window | `64` |
| `kv_cache_scale_frames` | Count of scale frames that remain permanently resident | `8` |
| `kv_cache_cross_frame_special` | Retain special tokens from evicted frames for cross-frame attention | `True` |
| `kv_cache_include_scale_frames` | Include scale frames in the cache (disable to save memory) | `True` |
| `kv_cache_camera_only` | Restrict caching to camera tokens only (experimental) | `False` |

These parameters are passed directly to the `FlashInferKVCacheManager` during `GCTStream` initialization, allowing users to balance context retention against memory constraints.

## Practical Implementation Examples

### Initializing a Streaming Model with Custom Cache Settings

Configure the KV cache dimensions during model instantiation to optimize for your hardware constraints:

```python
import torch
from lingbot_map.models.gct_stream_window_v2 import GCTStream

model = GCTStream(
    img_size=518,
    patch_size=14,
    embed_dim=1024,
    kv_cache_sliding_window=128,   # Larger window retains more temporal context

    kv_cache_scale_frames=4,       # Reduce resident scale frames to save memory

    kv_cache_cross_frame_special=False,
    enable_3d_rope=True,           # Maintain temporal RoPE consistency

)
model.eval()

```

### Running Frame-by-Frame Inference

The high-level API automatically manages cache append and eviction operations:

```python

# Input shape: [B, S, 3, H, W] representing a video clip

imgs = torch.randn(1, 120, 3, 518, 518)   # 120-frame video

# Process with fixed-interval keyframe selection

predictions = model.inference_streaming(
    images=imgs,
    keyframe_interval=5,
    output_device=torch.device('cpu'),   # Offload to prevent GPU OOM

)

```

Internally, this method clears the cache via `clean_kv_cache()`, appends KV tensors for each frame, and triggers eviction according to the sliding window policy. When using flow-based keyframe selection, the system automatically handles deferred eviction and rollback operations.

### Manual Cache Control and Debugging

For advanced use cases requiring explicit cache manipulation:

```python

# Reset cache when switching to a new video sequence

model.clean_kv_cache()

# Process a frame without storing its KV (non-keyframe skipping)

model._set_skip_append(True)
frame_output = model.forward(frame_tensor)   # frame_tensor: [1, 1, 3, H, W]

model._set_skip_append(False)                # Re-enable caching

# Inspect cache statistics for debugging

stats = model.aggregator.kv_cache_manager.get_cache_stats(block_idx=0)
print(f"Frame count: {stats['frame_count']}, "
      f"Live pages: {stats['live_pages']}, "
      f"Scale pages: {stats['scale_pages']}")

```

Set the environment variable `LINGBOT_DEBUG_KV` to enable per-frame diagnostic logging via the internal `_log_kv_stats` helper method.

## Summary

- **Two-stream architecture:** Separates recyclable patch tokens from append-only special tokens to optimize memory layout.
- **Paged FlashInfer backend:** Delegates attention computation to optimized CUDA kernels while maintaining O(1) memory complexity.
- **Sliding-window eviction:** Automatically recycles old frame pages while preserving resident scale frames for stable context.
- **Deferred eviction API:** Supports intelligent keyframe selection through `_execute_deferred_eviction` and `_rollback_last_frame`, enabling flow-based filtering without cache pollution.
- **Configurable parameters:** Expose window sizes and scale frame counts through `GCTStream` initialization to balance context retention against GPU memory limits.

## Frequently Asked Questions

### What is the KV cache in LingBot-Map?

The KV cache is a paged memory buffer managed by `FlashInferKVCacheManager` that stores key and value tensors from previously processed video frames. It enables the causal transformer to attend to historical context without recomputing attention weights for every token in the sequence, reducing inference complexity from quadratic to linear with respect to sequence length.

### How does the sliding window eviction work?

The `evict_frames` method maintains a fixed-size window of recent frames (configured via `kv_cache_sliding_window`). When new frames are appended, the oldest window pages are returned to a free list while scale pages (the first `kv_cache_scale_frames` entries) remain permanently resident. This ensures constant memory usage during long video streams.

### What are scale frames and why are they never evicted?

Scale frames are the initial `kv_cache_scale_frames` entries in the patch stream that contain scale token embeddings. They remain permanently cached to provide stable bidirectional context for scale-related processing throughout the video. Unlike regular window frames, these pages are exempt from the `evict_frames` logic to maintain consistent normalization and reference points.

### How can I debug KV cache usage during inference?

Call `model.aggregator.kv_cache_manager.get_cache_stats(block_idx=0)` to retrieve current statistics including frame counts, page allocations, and free list status. Alternatively, set the `LINGBOT_DEBUG_KV` environment variable before running inference to enable automatic per-frame logging via the internal `_log_kv_stats` method in `GCTStream`.