# How LingBot-Map Achieves ~20 FPS Inference with Its KV Cache

> Discover how LingBot-Map achieves ~20 FPS inference using its stream-based KV cache, reusing attention activations and intelligently evicting stale data for efficient video analysis.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: performance
- Published: 2026-07-03

---

**LingBot-Map achieves real-time ~20 FPS inference by streaming video frames through a sliding-window key-value (KV) cache that reuses attention activations across frames, evicting stale entries while selectively retaining key-frames based on optical flow or fixed intervals.**

The LingBot-Map repository (`Robbyant/lingbot-map`) implements a high-throughput visual localization architecture that processes continuous video streams without recomputing past attention weights. By leveraging a **LingBot-Map KV cache** combined with FlashInfer-optimized CUDA kernels, the model maintains constant-time per-frame latency regardless of sequence length. This design enables the system to run at approximately 20 frames per second on a single GPU while keeping memory usage bounded.

## Understanding the KV Cache Architecture

The foundation of LingBot-Map's real-time performance lies in how it caches attention states across video frames.

### Transformer-Level Caching

In [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py), each Transformer attention layer stores the *key* and *value* tensors from previously processed frames in GPU memory. When a new frame arrives, the model computes only the **query** tensor, then concatenates it with the cached KV tensors from prior frames (lines 58-70). This allows the attention mechanism to attend to the full temporal history without re-computing activations for past frames. The cache persists across forward passes, enabling **O(1) per-frame computation** rather than O(N) scaling with sequence length.

### Sliding-Window Eviction Policy

To prevent unbounded memory growth, the implementation enforces a strict limit on cache size. The `kv_cache_sliding_window` parameter (default 64) defines the maximum number of past frames retained in GPU memory (lines 223-227). When the cache reaches capacity, the FlashInfer manager (or SDPA fallback) evicts the oldest KV pages automatically.

Crucially, the system can preserve special tokens from evicted frames via `kv_cache_cross_frame_special` (lines 225-226). This ensures that important boundary markers remain accessible even after their corresponding frame data is removed from the cache.

## Key-Frame Selection Strategies

Not every frame contributes equally to the cache. LingBot-Map implements two distinct modes for selecting which frames persist in the KV cache.

### Fixed-Interval Key-Frames

In the simplest configuration, the model uses a fixed sampling interval specified by `keyframe_interval`. Every N-th frame is designated as a key-frame and stored in the cache, while intermediate frames are processed but discarded. For non-key-frames, the internal flag `_skip_append` is set to `True`, preventing the KV tensors from being appended to the persistent cache after the forward pass (lines 404-409, 4-10). This reduces cache pressure by only retaining 1/N of all frames while maintaining temporal continuity.

### Flow-Based Adaptive Key-Frames

For dynamic scenes, the system computes optical flow magnitude on-the-fly using `_compute_flow_magnitude`. If the motion between frames exceeds `flow_threshold`, the current frame becomes a key-frame and its KV cache is preserved. Otherwise, the system invokes `_rollback_last_frame` to undo the cache append operation after a deferred-eviction pass (lines 26-34, 52-62). This **adaptive key-frame selection** concentrates cache capacity on high-motion segments while skipping redundant static frames.

## Deferred Eviction and Rollback Mechanism

Handling non-key-frames requires careful cache management to prevent temporary entries from polluting the persistent store. The implementation uses a deferred eviction strategy to buffer this decision.

When processing a frame that might be discarded, the model first calls `_set_defer_eviction(True)` to prevent the KV manager from discarding any pages during the forward pass (lines 25-33). After inference completes, the system evaluates whether the frame qualifies as a key-frame. If not, it executes `_rollback_last_frame` to revert the cache to its previous state, then re-enables normal eviction (lines 49-68). This **atomic commit/rollback pattern** ensures that only selected frames modify the persistent cache, maintaining the integrity of the sliding window.

## Memory Optimization and FlashInfer Backend

Achieving 20 FPS requires not just algorithmic efficiency but hardware-optimized memory management.

### FlashInfer CUDA Kernels

By default, LingBot-Map sets `use_sdpa=False` and utilizes the FlashInfer backend via [`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py) and [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py). FlashInfer stores KV pages in a compact, paged format and performs batched look-ups using highly optimized CUDA kernels (lines 124-130). This implementation allows the cache to be consulted at frequencies exceeding 200 Hz, providing the low-latency foundation necessary for the overall 20 FPS throughput target.

### Streaming Inference Loop

The high-level `inference_streaming` method in [`gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream_window_v2.py) orchestrates the memory-efficient pipeline. Each frame is moved to the GPU, processed through the KV cache, and then immediately transferred to the user-specified `output_device` (typically CPU) (lines 69-84). The method calls `torch.cuda.empty_cache()` after each frame to prevent memory fragmentation and OOM errors (lines 91-100). This **per-frame memory discipline** ensures that the model can process arbitrarily long videos without accumulating GPU memory overhead.

## Implementation Example

The following example demonstrates how to configure the KV cache for streaming inference:

```python
import torch
from lingbot_map.models.gct_stream_window_v2 import GCTStream

# Initialize with KV cache parameters

model = GCTStream(
    pretrained_path="dino_vitl14_reg.pth",
    enable_stream_inference=True,
    kv_cache_sliding_window=64,          # Retain up to 64 past frames

    kv_cache_cross_frame_special=True,   # Preserve special tokens

    enable_3d_rope=False,
)

model.eval()
model.to("cuda")

# Simulate a 200-frame video sequence

video = torch.rand(200, 3, 518, 518).float() / 255.0

# Run streaming inference with fixed-interval keyframes

predictions = model.inference_streaming(
    images=video,                # [S, C, H, W]

    keyframe_interval=4,        # Store every 4th frame in cache

    output_device=torch.device("cpu"),
    flow_threshold=0.0,         # Disable flow-based selection

)

# Output shape: [1, 200, 9] for pose estimates

print(predictions["pose_enc"].shape)   # torch.Size([1, 200, 9])

```

In this configuration, the model processes all 200 frames but only appends 50 key-frames to the KV cache, reducing memory usage by 75% while maintaining the temporal context necessary for accurate pose estimation.

## Summary

- **LingBot-Map KV cache** reuses attention keys and values across video frames, eliminating redundant computation in the Transformer layers.
- A **sliding-window eviction policy** (default 64 frames) caps memory usage by discarding old entries while optionally preserving special tokens.
- **Key-frame selection** operates in either fixed-interval or optical-flow adaptive modes, concentrating cache capacity on informative frames.
- **Deferred eviction and rollback** ensure that non-key-frames do not pollute the persistent cache, implemented via `_set_defer_eviction` and `_rollback_last_frame`.
- **FlashInfer backend** provides hardware-optimized page management, enabling cache consultation at >200 Hz.
- The **streaming inference loop** moves predictions to CPU immediately and clears GPU memory after each frame, preventing OOM errors during long sequences.

## Frequently Asked Questions

### How does the KV cache prevent memory overflow during long videos?

The KV cache implements a **sliding-window eviction policy** defined by `kv_cache_sliding_window` (default 64 frames). When the cache exceeds this limit, the oldest KV pages are automatically evicted by the FlashInfer manager (lines 223-227). Additionally, the deferred eviction mechanism ensures that only selected key-frames are committed to the cache, while non-key-frames are rolled back immediately after processing.

### What is the difference between fixed-interval and flow-based key-frame selection?

**Fixed-interval mode** stores every N-th frame (controlled by `keyframe_interval`) regardless of content, using the `_skip_append` flag to discard non-key-frames (lines 404-409). **Flow-based mode** computes optical flow magnitude via `_compute_flow_magnitude` and only caches frames where motion exceeds `flow_threshold`. This adaptive approach concentrates limited cache capacity on high-motion segments while skipping static frames (lines 26-34).

### Why does LingBot-Map use FlashInfer instead of standard SDPA?

FlashInfer provides **paged KV cache management** and optimized CUDA kernels that perform batched lookups significantly faster than standard Scaled Dot-Product Attention (SDPA). As implemented in [`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py), FlashInfer enables cache consultation rates exceeding 200 Hz (lines 124-130), which is essential for achieving the overall ~20 FPS inference target while maintaining a manageable memory footprint.

### How does the rollback mechanism work for non-key-frames?

When processing frames that might be discarded, the system enters a deferred state by calling `_set_defer_eviction(True)` to prevent cache modifications during the forward pass (lines 25-33). If the frame is determined to be a non-key-frame after inference, the model executes `_rollback_last_frame` to revert the cache to its previous state, then re-enables normal eviction (lines 49-68). This atomic approach ensures cache consistency without requiring complex transaction logic.