FlashInfer Backend in LingBot-Map: Accelerating Streaming Inference with Paged KV Cache

The FlashInfer backend delivers ~20 FPS streaming inference and maintains bounded GPU memory usage for sequences exceeding 10,000 frames through a specialized paged KV cache implementation, serving as the default attention mechanism in LingBot-Map.

LingBot-Map is an open-source spatial reasoning model developed by Robbyant for 3D reconstruction from long video sequences. The repository implements a dual-backend attention architecture where the FlashInfer backend functions as the production-grade engine, while PyTorch's native SDPA serves as a compatibility fallback for unsupported hardware.

Technical Architecture of the FlashInfer Backend

Paged KV Cache Implementation

The FlashInfer backend's core innovation lies in its two-stream paged KV cache design implemented in lingbot_map/layers/flashinfer_cache.py. Rather than storing the full key-value history in contiguous GPU memory—which causes O(N) memory growth—the cache partitions data into three distinct storage strategies:

• Scale pages: Permanently resident memory pages that persist across the entire sequence
• Sliding-window patch pages: Rotary-evicted pages that maintain a fixed-size local context
• Append-only special stream: Dedicated storage for special tokens that accumulate without bound

This architecture allows LingBot-Map to process inputs of arbitrary length without exhausting VRAM, making it suitable for streaming applications that handle 10,000+ frames.

JIT-Compiled CUDA Execution

The backend packs patch and special tokens into CUDA pages and executes the complete Q-K-V attention computation in a single JIT-compiled kernel. This eliminates Python-to-CUDA dispatch overhead that typically bottlenecks transformer inference. When combined with PyTorch compilation (--compile flag), the system achieves ~20 FPS throughput on 518×378 resolution inputs according to the repository benchmarks.

FlashInfer vs SDPA: Backend Selection Guide

LingBot-Map ships with two mutually exclusive attention implementations:

FlashInfer (default): Requires a CUDA-enabled GPU and the flashinfer-python package. Uses paged KV caching with BF16 precision for optimal speed and memory efficiency. Ideal for production streaming reconstruction and long-sequence processing.

SDPA (PyTorch native): Falls back to torch.nn.functional.scaled_dot_product_attention in FP32 or BF16. Stores the complete KV matrix for every frame, causing linear memory growth with sequence length. Suitable for CPU-only environments, older GPUs lacking compute capability SM 70+, or debugging scenarios requiring numerical precision checks.

The runtime selection occurs in lingbot_map/layers/attention.py via the use_flashinfer boolean flag, which the high-level streaming logic in lingbot_map/aggregator/stream.py sets using use_flashinfer = not use_sdpa.

Installation and Usage

Install the FlashInfer dependency to enable the default backend:

# Install FlashInfer from PyPI

pip install --index-url https://pypi.org/simple flashinfer-python

# Optional: CUDA-specific JIT cache for faster initial compilation

pip install flashinfer-jit-cache -f https://flashinfer.ai/whl/cu128/flashinfer-jit-cache/

Run the interactive demo with FlashInfer automatically selected:

python demo.py \
    --model_path /path/to/lingbot-map-long.pt \
    --image_folder example/courthouse \
    --mask_sky \
    --compile

Force the SDPA backend when FlashInfer is unavailable:

python demo.py \
    --model_path /path/to/lingbot-map-long.pt \
    --image_folder example/courthouse \
    --use_sdpa

Memory Management and Debugging

The FlashInferKVCacheManager class in flashinfer_cache.py accepts a force_fp32=True parameter for numerical accuracy debugging. When enabled, this runs the paged attention in full precision rather than the default BF16, useful for validating model correctness during development.

For memory profiling, the repository includes scripts/benchmark_gct_memory.py, which demonstrates the bounded memory characteristics of FlashInfer compared to the unbounded growth exhibited by SDPA over long sequences.

Performance Benchmarking

Compare backend performance using the provided profiling utilities:

# Benchmark FlashInfer with compilation

python gct_profile.py \
    --backend flashinfer \
    --dtype bf16 \
    --compile \
    --model_path /path/to/lingbot-map.pt

# Benchmark SDPA fallback

python gct_profile.py \
    --backend sdpa \
    --dtype bf16 \
    --model_path /path/to/lingbot-map.pt

The gct_profile.py script reports per-step latency and peak GPU memory allocation, quantifying the trade-off between FlashInfer's speed and SDPA's broader hardware compatibility.

Summary

• FlashInfer is the default backend in LingBot-Map, providing production-grade streaming inference through paged KV caching in lingbot_map/layers/flashinfer_cache.py.
• Automatic fallback to SDPA occurs when FlashInfer import fails or when explicitly requested via --use_sdpa in lingbot_map/layers/attention.py.
• Paged attention architecture splits the KV cache into scale, sliding-window, and special-token streams, preventing O(N) memory growth for long sequences.
• ~20 FPS performance is achievable on 518×378 inputs when combining FlashInfer with PyTorch compilation.
• Installation requires the flashinfer-python wheel and a CUDA-capable GPU with compute capability SM 70 or higher.

Frequently Asked Questions

How does FlashInfer prevent out-of-memory errors during long video processing?

FlashInfer implements a paged KV cache that splits memory into scale pages (permanent), sliding-window patch pages (rotary-evicted), and an append-only special stream. This design keeps GPU memory usage bounded regardless of frame count, allowing processing of 10,000+ frame sequences that would exhaust VRAM under standard attention mechanisms.

What GPU requirements are necessary for FlashInfer to function?

FlashInfer requires any CUDA-enabled NVIDIA GPU with compute capability SM 70 or newer (Turing architecture and later). The backend JIT-compiles kernels at first use, requiring the flashinfer-python package to be installed. Pre-Turing GPUs or CPU-only environments must use the --use_sdpa fallback flag.

Why would I choose SDPA over FlashInfer despite the performance difference?

Select SDPA when running on unsupported hardware (pre-SM 70 GPUs), in offline air-gapped environments without FlashInfer wheels, or when debugging numerical precision issues. SDPA supports FP32 mode via force_fp32=True for validation, whereas FlashInfer is optimized for BF16 production inference.

Where does the backend selection logic reside in the codebase?

The selection occurs in lingbot_map/layers/attention.py, which checks the use_flashinfer flag and handles import errors gracefully. The lingbot_map/aggregator/stream.py module coordinates this setting based on CLI arguments, while lingbot_map/layers/flashinfer_cache.py contains the actual paged cache implementation and CUDA kernel interfaces.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}