# How KV Cache Is Handled in LingBot-Map's Streaming Aggregator

> Explore how Robbyant/lingbot-map's AggregatorStream efficiently handles KV cache with a paged key-value cache and dual-backend support for high-performance inference and SDPA compatibility.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: internals
- Published: 2026-07-07

---

**The `AggregatorStream` class in Robbyant/lingbot-map implements a paged key-value cache with dual-backend support, lazily initializing a `FlashInferKVCacheManager` for high-performance inference while maintaining a Python dictionary fallback for SDPA compatibility, enabling efficient causal attention across video frames without recomputation.**

The repository Robbyant/lingbot-map provides a streaming aggregator designed for real-time SLAM tasks that processes video frames sequentially. Understanding how KV cache is handled in [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py) is critical for optimizing GPU memory usage and maintaining low latency during streaming inference.

## KV Cache Initialization and Lazy Allocation

During construction, the `AggregatorStream` class stores KV cache hyperparameters but defers actual memory allocation until the first forward pass. In `__init__` (lines 34–49), the class captures configuration options including `kv_cache_sliding_window`, `kv_cache_scale_frames`, and `kv_cache_camera_only`. These parameters control eviction policies and memory retention strategies.

The method `_init_kv_cache()` (lines 79–86) creates placeholder structures rather than allocating tensors immediately. For the FlashInfer backend, it prepares an empty manager slot; for the SDPA fallback, it initializes an empty Python dictionary. This lazy design prevents unnecessary memory allocation when the model loads but before streaming begins.

## Lazy FlashInfer Manager Creation

When the first causal block requires cache access, `_get_flashinfer_manager()` (lines 198–229) dynamically imports `FlashInferKVCacheManager` from [`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py). The manager initializes with the model's depth, maximum frame capacity, tokens per frame, and the KV cache options defined during construction.

The manager allocates page-wise tensors internally and tracks frame counts, applying the sliding-window eviction policy defined by `kv_cache_sliding_window`. This approach retains only the most recent *N* frames while optionally preserving special scale frames and camera tokens according to the configuration flags.

## Cache Reset Between Video Sequences

Before processing a new video sequence, `clean_kv_cache()` (lines 236–248) clears the cache state to prevent cross-contamination between unrelated streams. For FlashInfer backends, the method calls the manager's `reset()` method; for SDPA mode, it zeroes out dictionary entries. The method also resets the frame counter `total_frames_processed` and clears cached 3-D RoPE positional embeddings.

## Forward Pass Execution with Dual Backend Support

The core inference logic in `_process_causal_stream` (lines 495–525) branches based on `self.use_sdpa`:

- **SDPA mode** (`True`): Passes the dictionary-based `self.kv_cache` to each `global_block`. The blocks update the dictionary with new keys and values during each iteration.
- **FlashInfer mode** (`False`): Retrieves the lazily-created manager via `_get_flashinfer_manager()` and passes it to the blocks. The manager handles page allocation and eviction automatically.

Both branches increment `self.total_frames_processed` only for the first block group of a sequence, ensuring accurate frame counting while allowing the cache to accumulate context across the sliding window.

## KV Cache Configuration Parameters

The following constructor arguments control cache behavior:

- **`kv_cache_sliding_window`**: Defines the number of recent frames retained; older frames undergo eviction to limit memory growth.
- **`kv_cache_scale_frames`**: Specifies how many scale tokens remain permanently cached for metric-scale estimation.
- **`kv_cache_cross_frame_special`**: Preserves special tokens (camera/register) from evicted frames when enabled.
- **`kv_cache_include_scale_frames`**: Determines whether scale frames populate the KV cache at all.
- **`kv_cache_camera_only`**: Restricts retention to camera tokens only during eviction, yielding additional memory savings.

These parameters enable developers to balance latency, GPU memory constraints, and accuracy for real-time SLAM applications.

## Practical Implementation Examples

The following example demonstrates FlashInfer backend initialization with a 64-frame sliding window:

```python
import torch
from lingbot_map.aggregator.stream import AggregatorStream

# Initialize with FlashInfer backend and moderate cache size

agg = AggregatorStream(
    sliding_window_size=32,
    kv_cache_sliding_window=64,
    kv_cache_scale_frames=8,
    use_sdpa=False,
    embed_dim=1024,
    depth=12,
    patch_size=16,
    img_size=640,
)

# Process dummy frame tokens (batch=1, sequence=1)

frames = torch.randn(1, 1, (640 // 16) ** 2 + agg.num_special_tokens, 1024)
out, _ = agg.forward(frames, pos=None)

# Reset cache before new video sequence

agg.clean_kv_cache()

```

To use the pure-PyTorch SDPA fallback with dictionary-based caching:

```python
agg_sdpa = AggregatorStream(
    use_sdpa=True,
    kv_cache_sliding_window=64,
    kv_cache_scale_frames=8,
    embed_dim=1024,
    depth=12,
    patch_size=16,
    img_size=640,
)

# Forward pass uses Python dict storage accessible via agg_sdpa.kv_cache

```

## Summary

- **Dual-backend architecture**: The `AggregatorStream` supports both FlashInfer (paged, high-performance) and SDPA (dictionary-based, dependency-free) KV cache implementations.
- **Lazy initialization**: The `FlashInferKVCacheManager` instantiates only when needed, while SDPA caches initialize as empty dictionaries in `_init_kv_cache()`.
- **Sliding-window eviction**: The cache automatically retains only the most recent frames defined by `kv_cache_sliding_window`, with special handling for scale and camera tokens.
- **Sequence isolation**: `clean_kv_cache()` ensures complete state reset between unrelated video streams, preventing context leakage.
- **Configurable retention**: Five distinct parameters control memory retention policies, enabling optimization for specific SLAM workloads.

## Frequently Asked Questions

### How does LingBot-Map decide between FlashInfer and SDPA for KV cache management?

The `AggregatorStream` checks `self.use_sdpa` at runtime. When `True`, it uses a Python dictionary (`self.kv_cache`) compatible with PyTorch's native scaled-dot-product attention. When `False`, it lazily initializes `FlashInferKVCacheManager` via `_get_flashinfer_manager()` to provide paged memory management and optimized attention kernels.

### What triggers the sliding window eviction in the KV cache?

The `FlashInferKVCacheManager` automatically applies eviction when the number of processed frames exceeds `kv_cache_sliding_window`. Older frames are removed from the page table, though special tokens may be retained if `kv_cache_cross_frame_special` or `kv_cache_camera_only` are enabled. The SDPA backend implements similar logic within the attention blocks themselves.

### Why is the KV cache initialized lazily rather than in `__init__`?

Lazy initialization in `_get_flashinfer_manager()` prevents unnecessary GPU memory allocation during model loading and allows the system to import heavy dependencies like FlashInfer only when required. This design reduces startup time and memory footprint for applications that may use the SDPA fallback exclusively.

### Where is the cache cleared when processing a new video sequence?

The `clean_kv_cache()` method (lines 236–248) handles cache reset by calling `reset()` on the FlashInfer manager or clearing the SDPA dictionary. It also resets `total_frames_processed` and clears cached 3-D RoPE positions, ensuring no temporal state persists between sequences.