# FlashInfer Backend in LingBot-Map: Accelerating Streaming Inference with Paged KV Cache

> Discover how FlashInfer backend in LingBot-Map accelerates streaming inference with its paged KV cache, achieving ~20 FPS and bounded GPU memory for long sequences.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: performance
- Published: 2026-07-23

---

**The FlashInfer backend delivers ~20 FPS streaming inference and maintains bounded GPU memory usage for sequences exceeding 10,000 frames through a specialized paged KV cache implementation, serving as the default attention mechanism in LingBot-Map.**

LingBot-Map is an open-source spatial reasoning model developed by Robbyant for 3D reconstruction from long video sequences. The repository implements a dual-backend attention architecture where the **FlashInfer backend** functions as the production-grade engine, while PyTorch's native SDPA serves as a compatibility fallback for unsupported hardware.

## Technical Architecture of the FlashInfer Backend

### Paged KV Cache Implementation

The FlashInfer backend's core innovation lies in its two-stream paged KV cache design implemented in [`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py). Rather than storing the full key-value history in contiguous GPU memory—which causes O(N) memory growth—the cache partitions data into three distinct storage strategies:

- **Scale pages**: Permanently resident memory pages that persist across the entire sequence
- **Sliding-window patch pages**: Rotary-evicted pages that maintain a fixed-size local context
- **Append-only special stream**: Dedicated storage for special tokens that accumulate without bound

This architecture allows LingBot-Map to process inputs of arbitrary length without exhausting VRAM, making it suitable for streaming applications that handle 10,000+ frames.

### JIT-Compiled CUDA Execution

The backend packs patch and special tokens into CUDA pages and executes the complete Q-K-V attention computation in a **single JIT-compiled kernel**. This eliminates Python-to-CUDA dispatch overhead that typically bottlenecks transformer inference. When combined with PyTorch compilation (`--compile` flag), the system achieves **~20 FPS** throughput on 518×378 resolution inputs according to the repository benchmarks.

## FlashInfer vs SDPA: Backend Selection Guide

LingBot-Map ships with two mutually exclusive attention implementations:

**FlashInfer** (default): Requires a CUDA-enabled GPU and the `flashinfer-python` package. Uses paged KV caching with BF16 precision for optimal speed and memory efficiency. Ideal for production streaming reconstruction and long-sequence processing.

**SDPA** (PyTorch native): Falls back to `torch.nn.functional.scaled_dot_product_attention` in FP32 or BF16. Stores the complete KV matrix for every frame, causing linear memory growth with sequence length. Suitable for CPU-only environments, older GPUs lacking compute capability SM 70+, or debugging scenarios requiring numerical precision checks.

The runtime selection occurs in [`lingbot_map/layers/attention.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/attention.py) via the `use_flashinfer` boolean flag, which the high-level streaming logic in [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py) sets using `use_flashinfer = not use_sdpa`.

## Installation and Usage

Install the FlashInfer dependency to enable the default backend:

```bash

# Install FlashInfer from PyPI

pip install --index-url https://pypi.org/simple flashinfer-python

# Optional: CUDA-specific JIT cache for faster initial compilation

pip install flashinfer-jit-cache -f https://flashinfer.ai/whl/cu128/flashinfer-jit-cache/

```

Run the interactive demo with FlashInfer automatically selected:

```bash
python demo.py \
    --model_path /path/to/lingbot-map-long.pt \
    --image_folder example/courthouse \
    --mask_sky \
    --compile

```

Force the SDPA backend when FlashInfer is unavailable:

```bash
python demo.py \
    --model_path /path/to/lingbot-map-long.pt \
    --image_folder example/courthouse \
    --use_sdpa

```

## Memory Management and Debugging

The `FlashInferKVCacheManager` class in [`flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/flashinfer_cache.py) accepts a `force_fp32=True` parameter for numerical accuracy debugging. When enabled, this runs the paged attention in full precision rather than the default BF16, useful for validating model correctness during development.

For memory profiling, the repository includes [`scripts/benchmark_gct_memory.py`](https://github.com/Robbyant/lingbot-map/blob/main/scripts/benchmark_gct_memory.py), which demonstrates the bounded memory characteristics of FlashInfer compared to the unbounded growth exhibited by SDPA over long sequences.

## Performance Benchmarking

Compare backend performance using the provided profiling utilities:

```bash

# Benchmark FlashInfer with compilation

python gct_profile.py \
    --backend flashinfer \
    --dtype bf16 \
    --compile \
    --model_path /path/to/lingbot-map.pt

# Benchmark SDPA fallback

python gct_profile.py \
    --backend sdpa \
    --dtype bf16 \
    --model_path /path/to/lingbot-map.pt

```

The [`gct_profile.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_profile.py) script reports per-step latency and peak GPU memory allocation, quantifying the trade-off between FlashInfer's speed and SDPA's broader hardware compatibility.

## Summary

- **FlashInfer is the default backend** in LingBot-Map, providing production-grade streaming inference through paged KV caching in [`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py).
- **Automatic fallback** to SDPA occurs when FlashInfer import fails or when explicitly requested via `--use_sdpa` in [`lingbot_map/layers/attention.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/attention.py).
- **Paged attention architecture** splits the KV cache into scale, sliding-window, and special-token streams, preventing O(N) memory growth for long sequences.
- **~20 FPS performance** is achievable on 518×378 inputs when combining FlashInfer with PyTorch compilation.
- **Installation requires** the `flashinfer-python` wheel and a CUDA-capable GPU with compute capability SM 70 or higher.

## Frequently Asked Questions

### How does FlashInfer prevent out-of-memory errors during long video processing?

FlashInfer implements a paged KV cache that splits memory into scale pages (permanent), sliding-window patch pages (rotary-evicted), and an append-only special stream. This design keeps GPU memory usage bounded regardless of frame count, allowing processing of 10,000+ frame sequences that would exhaust VRAM under standard attention mechanisms.

### What GPU requirements are necessary for FlashInfer to function?

FlashInfer requires any CUDA-enabled NVIDIA GPU with compute capability SM 70 or newer (Turing architecture and later). The backend JIT-compiles kernels at first use, requiring the `flashinfer-python` package to be installed. Pre-Turing GPUs or CPU-only environments must use the `--use_sdpa` fallback flag.

### Why would I choose SDPA over FlashInfer despite the performance difference?

Select SDPA when running on unsupported hardware (pre-SM 70 GPUs), in offline air-gapped environments without FlashInfer wheels, or when debugging numerical precision issues. SDPA supports FP32 mode via `force_fp32=True` for validation, whereas FlashInfer is optimized for BF16 production inference.

### Where does the backend selection logic reside in the codebase?

The selection occurs in [`lingbot_map/layers/attention.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/attention.py), which checks the `use_flashinfer` flag and handles import errors gracefully. The [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py) module coordinates this setting based on CLI arguments, while [`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py) contains the actual paged cache implementation and CUDA kernel interfaces.