When to Use the FlashInfer Backend for LingBot-Map: Optimization Guide

Use the FlashInfer backend for LingBot-Map when running on CUDA-enabled GPUs to achieve ~20 FPS streaming inference with bounded memory usage via paged KV caching, while falling back to SDPA only for debugging or unsupported hardware.

LingBot-Map (Robbyant/lingbot-map) ships with two attention backends optimized for different deployment scenarios. Understanding when to activate the FlashInfer backend versus the PyTorch SDPA fallback is critical for optimizing throughput and preventing out-of-memory errors during long-sequence streaming reconstruction.

What Is the FlashInfer Backend?

The FlashInfer backend is a paged KV-cache implementation that packs patch and special tokens into CUDA pages and executes the entire attention computation in a single JIT-compiled kernel. According to the source code in lingbot_map/layers/flashinfer_cache.py, this design splits the cache into three distinct streams:

• Scale pages: Always resident in memory
• Sliding-window patch pages: Rotating buffer for visual tokens
• Append-only special stream: Dedicated path for special tokens

This architecture prevents the O(N) memory growth typical of standard attention mechanisms, allowing the model to process 10,000+ frames without exhausting VRAM.

When Should You Use the FlashInfer Backend?

Production Streaming Inference

Use FlashInfer for all production-grade streaming reconstructions where latency matters. The kernel processes all Q-K-V interactions in a single CUDA call, eliminating Python-to-CUDA dispatch overhead. When compiled with --compile, the reference implementation achieves ~20 FPS on 518×378 inputs, making it suitable for interactive demos and real-time applications.

Long Sequence Processing

Choose FlashInfer when processing sequences exceeding thousands of frames. The paged KV cache design keeps memory bounded regardless of sequence length, unlike the SDPA backend which stores a full KV matrix for every frame.

Installation Requirements

Activate FlashInfer only after confirming:

1. You have a CUDA-enabled NVIDIA GPU (compute capability SM 70 or newer recommended)
2. You installed the flashinfer-python wheel:

pip install --index-url https://pypi.org/simple flashinfer-python

3. For faster first-use JIT compilation, optionally install:

pip install flashinfer-jit-cache -f https://flashinfer.ai/whl/cu128/flashinfer-jit-cache/

When to Use the SDPA Backend Instead

The SDPA (Scaled Dot Product Attention) backend serves as a fallback when FlashInfer is unavailable or unsuitable. Located in lingbot_map/layers/attention.py, the SDPA path uses torch.nn.functional.scaled_dot_product_attention without paged caching.

Hardware Limitations and Debugging

Switch to SDPA when:

• GPU incompatibility: Your device lacks the required compute capability (pre-Ada GPUs older than SM 70) or you are running on CPU-only hardware
• JIT compilation failures: FlashInfer's JIT fails in offline environments or restricted containers
• Numerical verification: You need FP32 precision checks by setting force_fp32=True in FlashInferKVCacheManager
• Quick prototyping: Running short test sequences where memory overhead is irrelevant

Explicitly force SDPA via the CLI flag:

python demo.py \
    --model_path /path/to/lingbot-map-long.pt \
    --image_folder example/courthouse \
    --use_sdpa

Implementation Details From the Source Code

The backend selection logic is implemented across three key files:

lingbot_map/layers/flashinfer_cache.py: Implements the FlashInferKVCacheManager class managing the two-stream paged cache (scale and patch/special tokens). This file handles the memory-efficient paging strategy that bounds GPU memory usage.

lingbot_map/layers/attention.py: Contains the runtime dispatch logic that checks use_flashinfer availability. If FlashInfer cannot be imported, the code raises a clear error and automatically switches to the SDPA path.

lingbot_map/aggregator/stream.py: High-level streaming logic that sets use_flashinfer = not use_sdpa based on CLI arguments, coordinating the reconstruction pipeline with the selected backend.

scripts/benchmark_gct_memory.py: Provides memory profiling utilities that quantify the VRAM savings of FlashInfer's paged approach versus standard SDPA allocation.

How to Install and Configure FlashInfer

Standard Installation and Usage

Install the required wheel and run the interactive demo with automatic backend detection:

# Install FlashInfer (default backend)

pip install --index-url https://pypi.org/simple flashinfer-python

# Run with FlashInfer automatically selected

python demo.py \
    --model_path /path/to/lingbot-map-long.pt \
    --image_folder example/courthouse \
    --mask_sky \
    --compile

Benchmarking Both Backends

Compare performance characteristics using the provided profiling script:

# Benchmark FlashInfer

python gct_profile.py \
    --backend flashinfer \
    --dtype bf16 \
    --compile \
    --model_path /path/to/lingbot-map.pt

# Benchmark SDPA fallback

python gct_profile.py \
    --backend sdpa \
    --dtype bf16 \
    --model_path /path/to/lingbot-map.pt

The gct_profile.py script outputs per-step latency and peak GPU memory usage, making the speed-memory trade-off explicit.

Summary

• Use FlashInfer for all production streaming inference on CUDA GPUs to achieve ~20 FPS and bounded memory usage via paged KV caching.
• Install requirements include flashinfer-python and a GPU with compute capability SM 70+.
• Use SDPA only as a fallback for incompatible hardware, debugging (FP32 precision), or CPU-only environments by passing --use_sdpa.
• Key files governing this behavior are lingbot_map/layers/flashinfer_cache.py and lingbot_map/layers/attention.py.
• Benchmark both backends using gct_profile.py to quantify latency and memory differences for your specific hardware.

Frequently Asked Questions

Can I use FlashInfer on CPU-only machines?

No. FlashInfer requires a CUDA-enabled NVIDIA GPU and the flashinfer-python package. On CPU-only machines or systems without compatible CUDA drivers, use the SDPA backend by passing --use_sdpa to force PyTorch native attention.

How do I force LingBot-Map to use SDPA instead of FlashInfer?

Pass the --use_sdpa flag when running demo.py or set use_sdpa=True in the model configuration. The code in lingbot_map/aggregator/stream.py automatically sets use_flashinfer = not use_sdpa, routing attention computation to torch.nn.functional.scaled_dot_product_attention.

What GPU compute capability is required for FlashInfer?

FlashInfer works best on GPUs with compute capability SM 70 or newer (Ada architecture and above). Older GPUs may fail during JIT compilation or show degraded performance, in which case you should fall back to SDPA.

Does FlashInfer affect numerical precision compared to SDPA?

FlashInfer typically runs in BF16 for speed, while SDPA can be forced to FP32 for higher precision checks. If you need to debug numerical accuracy, set force_fp32=True in FlashInferKVCacheManager or switch to SDPA, which supports FP32 natively without additional configuration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}