How KV Cache Is Handled in LingBot-Map's Streaming Aggregator

The AggregatorStream class in Robbyant/lingbot-map implements a paged key-value cache with dual-backend support, lazily initializing a FlashInferKVCacheManager for high-performance inference while maintaining a Python dictionary fallback for SDPA compatibility, enabling efficient causal attention across video frames without recomputation.

The repository Robbyant/lingbot-map provides a streaming aggregator designed for real-time SLAM tasks that processes video frames sequentially. Understanding how KV cache is handled in lingbot_map/aggregator/stream.py is critical for optimizing GPU memory usage and maintaining low latency during streaming inference.

KV Cache Initialization and Lazy Allocation

During construction, the AggregatorStream class stores KV cache hyperparameters but defers actual memory allocation until the first forward pass. In __init__ (lines 34–49), the class captures configuration options including kv_cache_sliding_window, kv_cache_scale_frames, and kv_cache_camera_only. These parameters control eviction policies and memory retention strategies.

The method _init_kv_cache() (lines 79–86) creates placeholder structures rather than allocating tensors immediately. For the FlashInfer backend, it prepares an empty manager slot; for the SDPA fallback, it initializes an empty Python dictionary. This lazy design prevents unnecessary memory allocation when the model loads but before streaming begins.

Lazy FlashInfer Manager Creation

When the first causal block requires cache access, _get_flashinfer_manager() (lines 198–229) dynamically imports FlashInferKVCacheManager from lingbot_map/layers/flashinfer_cache.py. The manager initializes with the model's depth, maximum frame capacity, tokens per frame, and the KV cache options defined during construction.

The manager allocates page-wise tensors internally and tracks frame counts, applying the sliding-window eviction policy defined by kv_cache_sliding_window. This approach retains only the most recent N frames while optionally preserving special scale frames and camera tokens according to the configuration flags.

Cache Reset Between Video Sequences

Before processing a new video sequence, clean_kv_cache() (lines 236–248) clears the cache state to prevent cross-contamination between unrelated streams. For FlashInfer backends, the method calls the manager's reset() method; for SDPA mode, it zeroes out dictionary entries. The method also resets the frame counter total_frames_processed and clears cached 3-D RoPE positional embeddings.

Forward Pass Execution with Dual Backend Support

The core inference logic in _process_causal_stream (lines 495–525) branches based on self.use_sdpa:

• SDPA mode (True): Passes the dictionary-based self.kv_cache to each global_block. The blocks update the dictionary with new keys and values during each iteration.
• FlashInfer mode (False): Retrieves the lazily-created manager via _get_flashinfer_manager() and passes it to the blocks. The manager handles page allocation and eviction automatically.

Both branches increment self.total_frames_processed only for the first block group of a sequence, ensuring accurate frame counting while allowing the cache to accumulate context across the sliding window.

KV Cache Configuration Parameters

The following constructor arguments control cache behavior:

• kv_cache_sliding_window: Defines the number of recent frames retained; older frames undergo eviction to limit memory growth.
• kv_cache_scale_frames: Specifies how many scale tokens remain permanently cached for metric-scale estimation.
• kv_cache_cross_frame_special: Preserves special tokens (camera/register) from evicted frames when enabled.
• kv_cache_include_scale_frames: Determines whether scale frames populate the KV cache at all.
• kv_cache_camera_only: Restricts retention to camera tokens only during eviction, yielding additional memory savings.

These parameters enable developers to balance latency, GPU memory constraints, and accuracy for real-time SLAM applications.

Practical Implementation Examples

The following example demonstrates FlashInfer backend initialization with a 64-frame sliding window:

import torch
from lingbot_map.aggregator.stream import AggregatorStream

# Initialize with FlashInfer backend and moderate cache size

agg = AggregatorStream(
    sliding_window_size=32,
    kv_cache_sliding_window=64,
    kv_cache_scale_frames=8,
    use_sdpa=False,
    embed_dim=1024,
    depth=12,
    patch_size=16,
    img_size=640,
)

# Process dummy frame tokens (batch=1, sequence=1)

frames = torch.randn(1, 1, (640 // 16) ** 2 + agg.num_special_tokens, 1024)
out, _ = agg.forward(frames, pos=None)

# Reset cache before new video sequence

agg.clean_kv_cache()

To use the pure-PyTorch SDPA fallback with dictionary-based caching:

agg_sdpa = AggregatorStream(
    use_sdpa=True,
    kv_cache_sliding_window=64,
    kv_cache_scale_frames=8,
    embed_dim=1024,
    depth=12,
    patch_size=16,
    img_size=640,
)

# Forward pass uses Python dict storage accessible via agg_sdpa.kv_cache

Summary

• Dual-backend architecture: The AggregatorStream supports both FlashInfer (paged, high-performance) and SDPA (dictionary-based, dependency-free) KV cache implementations.
• Lazy initialization: The FlashInferKVCacheManager instantiates only when needed, while SDPA caches initialize as empty dictionaries in _init_kv_cache().
• Sliding-window eviction: The cache automatically retains only the most recent frames defined by kv_cache_sliding_window, with special handling for scale and camera tokens.
• Sequence isolation: clean_kv_cache() ensures complete state reset between unrelated video streams, preventing context leakage.
• Configurable retention: Five distinct parameters control memory retention policies, enabling optimization for specific SLAM workloads.

Frequently Asked Questions

How does LingBot-Map decide between FlashInfer and SDPA for KV cache management?

The AggregatorStream checks self.use_sdpa at runtime. When True, it uses a Python dictionary (self.kv_cache) compatible with PyTorch's native scaled-dot-product attention. When False, it lazily initializes FlashInferKVCacheManager via _get_flashinfer_manager() to provide paged memory management and optimized attention kernels.

What triggers the sliding window eviction in the KV cache?

The FlashInferKVCacheManager automatically applies eviction when the number of processed frames exceeds kv_cache_sliding_window. Older frames are removed from the page table, though special tokens may be retained if kv_cache_cross_frame_special or kv_cache_camera_only are enabled. The SDPA backend implements similar logic within the attention blocks themselves.

Why is the KV cache initialized lazily rather than in __init__?

Lazy initialization in _get_flashinfer_manager() prevents unnecessary GPU memory allocation during model loading and allows the system to import heavy dependencies like FlashInfer only when required. This design reduces startup time and memory footprint for applications that may use the SDPA fallback exclusively.

Where is the cache cleared when processing a new video sequence?

The clean_kv_cache() method (lines 236–248) handles cache reset by calling reset() on the FlashInfer manager or clearing the SDPA dictionary. It also resets total_frames_processed and clears cached 3-D RoPE positions, ensuring no temporal state persists between sequences.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}