FlashInfer Paged KV Cache Performance Benefits in LingBot-Map: Technical Implementation and Benchmarks

FlashInfer's paged KV cache reduces peak memory usage by 6× and delivers up to 2× faster attention compared to standard SDPA, enabling constant-memory streaming inference for long video sequences in LingBot-Map.

The Robbyant/lingbot-map repository implements a novel attention mechanism using FlashInfer's paged KV cache to overcome the memory bottlenecks of conventional Transformer inference. Unlike standard implementations that store full key-value tensors for all frames, LingBot-Map utilizes a two-stream paging system that maintains constant memory regardless of video length, processing long videos without out-of-memory errors.

Memory Architecture: Paged vs Standard Attention

Linear Memory Growth in Standard SDPA

Standard PyTorch scaled_dot_product_attention (SDPA) keeps the full [T × H × D] K and V tensors for all past frames in GPU memory, resulting in O(T) memory growth that linearly increases with frame count. For a 224×224 video with 320 frames, this approach consumes approximately 5 GB of peak KV memory, causing rapid degradation and out-of-memory errors after roughly 200 frames.

Constant Memory with Two-Stream Paging

In lingbot_map/layers/flashinfer_cache.py, LingBot-Map replaces monolithic tensors with a paged KV cache using two distinct streams:

• Patch stream: Stores recyclable pages for visual tokens (patches_per_frame = tokens_per_frame - num_special_tokens). Pages are reclaimed after exiting the scale and sliding-window frames, placing them on a free list for reuse.
• Special stream: Maintains append-only pages for camera, register, and scale tokens that are never evicted, ensuring persistent access to pose information across all frames.

This design bounds memory usage to max_patch_pages + max_special_pages, yielding a fixed ~850 MB allocation regardless of video duration—an 83% reduction compared to standard SDPA.

Computational Optimizations

Visible Page Table Attention

The FlashInferKVCacheManager builds a visible page table (scale + window + special) that tells the kernel exactly which pages contain valid tokens. In lingbot_map/layers/attention.py, the plan() method computes this table once per frame step on block 0, while run() executes on every layer using the same plan.

This optimization reduces the attention matrix from full history (e.g., 262 tokens) to only visible pages (e.g., ~70 tokens for a 64-frame window), eliminating wasted dot-products on evicted tokens.

Custom CUDA Kernel Implementation

FlashInfer's BatchPrefillWithPagedKVCacheWrapper streams pages directly from GPU cache using FP16/BF16 operations, avoiding the extra copies required by PyTorch's generic SDPA. The kernel attends only to pages listed in the visible page table, while the fallback --use_sdpa flag in lingbot_map/models/gct_stream_window_v2.py triggers standard SDPA when FlashInfer is unavailable.

Quantitative Performance Benchmarks

Metric	SDPA (Fallback)	FlashInfer (Paged KV)
Peak KV Memory (224×224, 320 frames)	~5 GB	~850 MB (constant)
Throughput	~12 fps	22 fps (1.8× speed-up)
Latency per Frame	83 ms	45 ms
Scalability	OOM after ~200 frames	No degradation

These measurements from benchmark/run.py demonstrate that the paged KV cache achieves constant-memory streaming inference, allowing processing of 320+ frame sequences without performance degradation—a scenario impossible with standard attention.

Implementation Details and Code Examples

Default FlashInfer Configuration

FlashInfer is the default backend unless explicitly disabled. The FlashInferKVCacheManager is lazily instantiated in lingbot_map/aggregator/stream.py:

from lingbot_map.models.gct_stream import LingBotMap

# Automatically creates FlashInferKVCacheManager

model = LingBotMap(pretrained=True)
model.eval()

# Video tensor: B=1, T=10, H*W=224*224, C=3

video = torch.randn(1, 10, 224*224, 3).cuda()

# Forward pass triggers plan() then run() for each layer

poses, recon = model(video)

The first forward call triggers the plan step via build_visible_page_table and compute_last_page_len, after which each frame only touches the visible pages.

SDPA Fallback Mode

For machines without FlashInfer installed, force standard attention:

python demo.py --use_sdpa

Or programmatically:

# Disables FlashInfer, uses SDPAAttention instead

model = LingBotMap(pretrained=True, use_sdpa=True)

This flag swaps the attention class in gct_stream_window_v2.py lines 229-235, ensuring compatibility across hardware configurations.

Monitoring Cache Statistics

Inspect runtime performance in lingbot_map/layers/flashinfer_cache.py using the built-in statistics method:

# After processing frames

stats = model.kv_cache_manager.get_cache_stats()
print(stats)

# Output: {'frame_count': 12, 'scale_pages': 8, 'live_pages': 4, 

#         'free_pages': 76, 'special_tokens': 72}

This reveals how the patch pool recycles pages while special tokens accumulate in dedicated pages.

Summary

FlashInfer's paged KV cache transforms LingBot-Map from a memory-heavy, quadratic-time system into a constant-memory, linear-time streaming architecture:

• Memory Efficiency: Fixed allocation of patch pages plus special pages replaces O(T) tensor growth, reducing VRAM usage by 6× on long videos.
• Compute Reduction: The visible page table ensures the attention kernel processes only relevant tokens, cutting unnecessary dot-products.
• Throughput Gains: Custom CUDA kernels and reduced data movement achieve up to 22 fps versus 12 fps with SDPA on RTX 3080 hardware.
• Graceful Degradation: Automatic fallback to SDPA ensures compatibility when FlashInfer is unavailable.

Frequently Asked Questions

What is the memory complexity difference between FlashInfer and standard attention?

Standard attention exhibits O(T) memory complexity, storing full KV tensors for all T frames and causing linear growth. FlashInfer's paged KV cache achieves O(1) constant memory by recycling patch pages outside the sliding window while keeping special tokens in fixed append-only pages.

How does the two-stream design work in LingBot-Map?

The architecture separates tokens into patch streams (visual features that can be evicted after leaving the attention window) and special streams (camera pose and register tokens that persist for the entire sequence). Patch pages are recycled via a free list in flashinfer_cache.py, while special pages are pre-allocated once and never evicted.

Can I use LingBot-Map without FlashInfer installed?

Yes. The repository automatically falls back to PyTorch's native SDPA when FlashInfer is unavailable by passing the --use_sdpa flag or setting use_sdpa=True in the model constructor. This fallback disables the paged cache and uses standard attention, though with higher memory usage and lower throughput.

What hardware benefits most from the paged KV cache implementation?

NVIDIA GPUs with CUDA support benefit significantly, particularly when processing long video sequences (200+ frames) where standard attention would hit out-of-memory errors. The FlashInfer kernels (FA2/FA3) optimized for FP16/BF16 operations deliver the best performance on modern GPUs like the RTX 3080 or A100, though the implementation works across all CUDA-capable devices.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}