How LingBot-Map Achieves ~20 FPS Inference with Its KV Cache

LingBot-Map achieves real-time ~20 FPS inference by streaming video frames through a sliding-window key-value (KV) cache that reuses attention activations across frames, evicting stale entries while selectively retaining key-frames based on optical flow or fixed intervals.

The LingBot-Map repository (Robbyant/lingbot-map) implements a high-throughput visual localization architecture that processes continuous video streams without recomputing past attention weights. By leveraging a LingBot-Map KV cache combined with FlashInfer-optimized CUDA kernels, the model maintains constant-time per-frame latency regardless of sequence length. This design enables the system to run at approximately 20 frames per second on a single GPU while keeping memory usage bounded.

Understanding the KV Cache Architecture

The foundation of LingBot-Map's real-time performance lies in how it caches attention states across video frames.

Transformer-Level Caching

In lingbot_map/models/gct_stream_window_v2.py, each Transformer attention layer stores the key and value tensors from previously processed frames in GPU memory. When a new frame arrives, the model computes only the query tensor, then concatenates it with the cached KV tensors from prior frames (lines 58-70). This allows the attention mechanism to attend to the full temporal history without re-computing activations for past frames. The cache persists across forward passes, enabling O(1) per-frame computation rather than O(N) scaling with sequence length.

Sliding-Window Eviction Policy

To prevent unbounded memory growth, the implementation enforces a strict limit on cache size. The kv_cache_sliding_window parameter (default 64) defines the maximum number of past frames retained in GPU memory (lines 223-227). When the cache reaches capacity, the FlashInfer manager (or SDPA fallback) evicts the oldest KV pages automatically.

Crucially, the system can preserve special tokens from evicted frames via kv_cache_cross_frame_special (lines 225-226). This ensures that important boundary markers remain accessible even after their corresponding frame data is removed from the cache.

Key-Frame Selection Strategies

Not every frame contributes equally to the cache. LingBot-Map implements two distinct modes for selecting which frames persist in the KV cache.

Fixed-Interval Key-Frames

In the simplest configuration, the model uses a fixed sampling interval specified by keyframe_interval. Every N-th frame is designated as a key-frame and stored in the cache, while intermediate frames are processed but discarded. For non-key-frames, the internal flag _skip_append is set to True, preventing the KV tensors from being appended to the persistent cache after the forward pass (lines 404-409, 4-10). This reduces cache pressure by only retaining 1/N of all frames while maintaining temporal continuity.

Flow-Based Adaptive Key-Frames

For dynamic scenes, the system computes optical flow magnitude on-the-fly using _compute_flow_magnitude. If the motion between frames exceeds flow_threshold, the current frame becomes a key-frame and its KV cache is preserved. Otherwise, the system invokes _rollback_last_frame to undo the cache append operation after a deferred-eviction pass (lines 26-34, 52-62). This adaptive key-frame selection concentrates cache capacity on high-motion segments while skipping redundant static frames.

Deferred Eviction and Rollback Mechanism

Handling non-key-frames requires careful cache management to prevent temporary entries from polluting the persistent store. The implementation uses a deferred eviction strategy to buffer this decision.

When processing a frame that might be discarded, the model first calls _set_defer_eviction(True) to prevent the KV manager from discarding any pages during the forward pass (lines 25-33). After inference completes, the system evaluates whether the frame qualifies as a key-frame. If not, it executes _rollback_last_frame to revert the cache to its previous state, then re-enables normal eviction (lines 49-68). This atomic commit/rollback pattern ensures that only selected frames modify the persistent cache, maintaining the integrity of the sliding window.

Memory Optimization and FlashInfer Backend

Achieving 20 FPS requires not just algorithmic efficiency but hardware-optimized memory management.

FlashInfer CUDA Kernels

By default, LingBot-Map sets use_sdpa=False and utilizes the FlashInfer backend via lingbot_map/layers/flashinfer_cache.py and lingbot_map/aggregator/stream.py. FlashInfer stores KV pages in a compact, paged format and performs batched look-ups using highly optimized CUDA kernels (lines 124-130). This implementation allows the cache to be consulted at frequencies exceeding 200 Hz, providing the low-latency foundation necessary for the overall 20 FPS throughput target.

Streaming Inference Loop

The high-level inference_streaming method in gct_stream_window_v2.py orchestrates the memory-efficient pipeline. Each frame is moved to the GPU, processed through the KV cache, and then immediately transferred to the user-specified output_device (typically CPU) (lines 69-84). The method calls torch.cuda.empty_cache() after each frame to prevent memory fragmentation and OOM errors (lines 91-100). This per-frame memory discipline ensures that the model can process arbitrarily long videos without accumulating GPU memory overhead.

Implementation Example

The following example demonstrates how to configure the KV cache for streaming inference:

import torch
from lingbot_map.models.gct_stream_window_v2 import GCTStream

# Initialize with KV cache parameters

model = GCTStream(
    pretrained_path="dino_vitl14_reg.pth",
    enable_stream_inference=True,
    kv_cache_sliding_window=64,          # Retain up to 64 past frames

    kv_cache_cross_frame_special=True,   # Preserve special tokens

    enable_3d_rope=False,
)

model.eval()
model.to("cuda")

# Simulate a 200-frame video sequence

video = torch.rand(200, 3, 518, 518).float() / 255.0

# Run streaming inference with fixed-interval keyframes

predictions = model.inference_streaming(
    images=video,                # [S, C, H, W]

    keyframe_interval=4,        # Store every 4th frame in cache

    output_device=torch.device("cpu"),
    flow_threshold=0.0,         # Disable flow-based selection

)

# Output shape: [1, 200, 9] for pose estimates

print(predictions["pose_enc"].shape)   # torch.Size([1, 200, 9])

In this configuration, the model processes all 200 frames but only appends 50 key-frames to the KV cache, reducing memory usage by 75% while maintaining the temporal context necessary for accurate pose estimation.

Summary

• LingBot-Map KV cache reuses attention keys and values across video frames, eliminating redundant computation in the Transformer layers.
• A sliding-window eviction policy (default 64 frames) caps memory usage by discarding old entries while optionally preserving special tokens.
• Key-frame selection operates in either fixed-interval or optical-flow adaptive modes, concentrating cache capacity on informative frames.
• Deferred eviction and rollback ensure that non-key-frames do not pollute the persistent cache, implemented via _set_defer_eviction and _rollback_last_frame.
• FlashInfer backend provides hardware-optimized page management, enabling cache consultation at >200 Hz.
• The streaming inference loop moves predictions to CPU immediately and clears GPU memory after each frame, preventing OOM errors during long sequences.

Frequently Asked Questions

How does the KV cache prevent memory overflow during long videos?

The KV cache implements a sliding-window eviction policy defined by kv_cache_sliding_window (default 64 frames). When the cache exceeds this limit, the oldest KV pages are automatically evicted by the FlashInfer manager (lines 223-227). Additionally, the deferred eviction mechanism ensures that only selected key-frames are committed to the cache, while non-key-frames are rolled back immediately after processing.

What is the difference between fixed-interval and flow-based key-frame selection?

Fixed-interval mode stores every N-th frame (controlled by keyframe_interval) regardless of content, using the _skip_append flag to discard non-key-frames (lines 404-409). Flow-based mode computes optical flow magnitude via _compute_flow_magnitude and only caches frames where motion exceeds flow_threshold. This adaptive approach concentrates limited cache capacity on high-motion segments while skipping static frames (lines 26-34).

Why does LingBot-Map use FlashInfer instead of standard SDPA?

FlashInfer provides paged KV cache management and optimized CUDA kernels that perform batched lookups significantly faster than standard Scaled Dot-Product Attention (SDPA). As implemented in lingbot_map/layers/flashinfer_cache.py, FlashInfer enables cache consultation rates exceeding 200 Hz (lines 124-130), which is essential for achieving the overall ~20 FPS inference target while maintaining a manageable memory footprint.

How does the rollback mechanism work for non-key-frames?

When processing frames that might be discarded, the system enters a deferred state by calling _set_defer_eviction(True) to prevent cache modifications during the forward pass (lines 25-33). If the frame is determined to be a non-key-frame after inference, the model executes _rollback_last_frame to revert the cache to its previous state, then re-enables normal eviction (lines 49-68). This atomic approach ensures cache consistency without requiring complex transaction logic.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}