# FlashInfer Paged KV Cache Performance Benefits in LingBot-Map: Technical Implementation and Benchmarks

> Discover FlashInfer's paged KV cache in LingBot-Map. Achieve 6x less memory and 2x faster attention for efficient long video inference. Learn the technical details and benchmarks.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: performance
- Published: 2026-06-29

---

**FlashInfer's paged KV cache reduces peak memory usage by 6× and delivers up to 2× faster attention compared to standard SDPA, enabling constant-memory streaming inference for long video sequences in LingBot-Map.**

The Robbyant/lingbot-map repository implements a novel attention mechanism using FlashInfer's paged KV cache to overcome the memory bottlenecks of conventional Transformer inference. Unlike standard implementations that store full key-value tensors for all frames, LingBot-Map utilizes a two-stream paging system that maintains constant memory regardless of video length, processing long videos without out-of-memory errors.

## Memory Architecture: Paged vs Standard Attention

### Linear Memory Growth in Standard SDPA

Standard PyTorch `scaled_dot_product_attention` (SDPA) keeps the full *[T × H × D]* K and V tensors for **all** past frames in GPU memory, resulting in **O(T)** memory growth that linearly increases with frame count. For a 224×224 video with 320 frames, this approach consumes approximately **5 GB** of peak KV memory, causing rapid degradation and out-of-memory errors after roughly 200 frames.

### Constant Memory with Two-Stream Paging

In [`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py), LingBot-Map replaces monolithic tensors with a **paged KV cache** using two distinct streams:

- **Patch stream**: Stores recyclable pages for visual tokens (`patches_per_frame = tokens_per_frame - num_special_tokens`). Pages are reclaimed after exiting the scale and sliding-window frames, placing them on a free list for reuse.
- **Special stream**: Maintains append-only pages for camera, register, and scale tokens that are never evicted, ensuring persistent access to pose information across all frames.

This design bounds memory usage to `max_patch_pages` + `max_special_pages`, yielding a fixed **~850 MB** allocation regardless of video duration—an 83% reduction compared to standard SDPA.

## Computational Optimizations

### Visible Page Table Attention

The `FlashInferKVCacheManager` builds a **visible page table** (scale + window + special) that tells the kernel exactly which pages contain valid tokens. In [`lingbot_map/layers/attention.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/attention.py), the `plan()` method computes this table once per frame step on block 0, while `run()` executes on every layer using the same plan.

This optimization reduces the attention matrix from full history (e.g., 262 tokens) to only visible pages (e.g., ~70 tokens for a 64-frame window), eliminating wasted dot-products on evicted tokens.

### Custom CUDA Kernel Implementation

FlashInfer's `BatchPrefillWithPagedKVCacheWrapper` streams pages directly from GPU cache using FP16/BF16 operations, avoiding the extra copies required by PyTorch's generic SDPA. The kernel attends **only** to pages listed in the visible page table, while the fallback `--use_sdpa` flag in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py) triggers standard SDPA when FlashInfer is unavailable.

## Quantitative Performance Benchmarks

| Metric | SDPA (Fallback) | FlashInfer (Paged KV) |
|--------|----------------|----------------------|
| **Peak KV Memory** (224×224, 320 frames) | ~5 GB | ~850 MB (constant) |
| **Throughput** | ~12 fps | **22 fps** (1.8× speed-up) |
| **Latency per Frame** | 83 ms | **45 ms** |
| **Scalability** | OOM after ~200 frames | No degradation |

These measurements from [`benchmark/run.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/run.py) demonstrate that the paged KV cache achieves **constant-memory streaming inference**, allowing processing of 320+ frame sequences without performance degradation—a scenario impossible with standard attention.

## Implementation Details and Code Examples

### Default FlashInfer Configuration

FlashInfer is the default backend unless explicitly disabled. The `FlashInferKVCacheManager` is lazily instantiated in [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py):

```python
from lingbot_map.models.gct_stream import LingBotMap

# Automatically creates FlashInferKVCacheManager

model = LingBotMap(pretrained=True)
model.eval()

# Video tensor: B=1, T=10, H*W=224*224, C=3

video = torch.randn(1, 10, 224*224, 3).cuda()

# Forward pass triggers plan() then run() for each layer

poses, recon = model(video)

```

The first forward call triggers the **plan** step via `build_visible_page_table` and `compute_last_page_len`, after which each frame only touches the visible pages.

### SDPA Fallback Mode

For machines without FlashInfer installed, force standard attention:

```bash
python demo.py --use_sdpa

```

Or programmatically:

```python

# Disables FlashInfer, uses SDPAAttention instead

model = LingBotMap(pretrained=True, use_sdpa=True)

```

This flag swaps the attention class in [`gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream_window_v2.py) lines 229-235, ensuring compatibility across hardware configurations.

### Monitoring Cache Statistics

Inspect runtime performance in [`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py) using the built-in statistics method:

```python

# After processing frames

stats = model.kv_cache_manager.get_cache_stats()
print(stats)

# Output: {'frame_count': 12, 'scale_pages': 8, 'live_pages': 4, 

#         'free_pages': 76, 'special_tokens': 72}

```

This reveals how the patch pool recycles pages while special tokens accumulate in dedicated pages.

## Summary

FlashInfer's paged KV cache transforms LingBot-Map from a memory-heavy, quadratic-time system into a constant-memory, linear-time streaming architecture:

- **Memory Efficiency**: Fixed allocation of patch pages plus special pages replaces O(T) tensor growth, reducing VRAM usage by 6× on long videos.
- **Compute Reduction**: The visible page table ensures the attention kernel processes only relevant tokens, cutting unnecessary dot-products.
- **Throughput Gains**: Custom CUDA kernels and reduced data movement achieve up to 22 fps versus 12 fps with SDPA on RTX 3080 hardware.
- **Graceful Degradation**: Automatic fallback to SDPA ensures compatibility when FlashInfer is unavailable.

## Frequently Asked Questions

### What is the memory complexity difference between FlashInfer and standard attention?

Standard attention exhibits **O(T)** memory complexity, storing full KV tensors for all T frames and causing linear growth. FlashInfer's paged KV cache achieves **O(1)** constant memory by recycling patch pages outside the sliding window while keeping special tokens in fixed append-only pages.

### How does the two-stream design work in LingBot-Map?

The architecture separates tokens into **patch streams** (visual features that can be evicted after leaving the attention window) and **special streams** (camera pose and register tokens that persist for the entire sequence). Patch pages are recycled via a free list in [`flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/flashinfer_cache.py), while special pages are pre-allocated once and never evicted.

### Can I use LingBot-Map without FlashInfer installed?

Yes. The repository automatically falls back to PyTorch's native SDPA when FlashInfer is unavailable by passing the `--use_sdpa` flag or setting `use_sdpa=True` in the model constructor. This fallback disables the paged cache and uses standard attention, though with higher memory usage and lower throughput.

### What hardware benefits most from the paged KV cache implementation?

NVIDIA GPUs with CUDA support benefit significantly, particularly when processing long video sequences (200+ frames) where standard attention would hit out-of-memory errors. The FlashInfer kernels (FA2/FA3) optimized for FP16/BF16 operations deliver the best performance on modern GPUs like the RTX 3080 or A100, though the implementation works across all CUDA-capable devices.