# When to Use the FlashInfer Backend for LingBot-Map: Optimization Guide

> Discover when to use the FlashInfer backend for LingBot-Map on CUDA GPUs for fast streaming inference. Optimize your setup for peak performance and bounded memory with paged KV caching.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: best-practices
- Published: 2026-07-06

---

**Use the FlashInfer backend for LingBot-Map when running on CUDA-enabled GPUs to achieve ~20 FPS streaming inference with bounded memory usage via paged KV caching, while falling back to SDPA only for debugging or unsupported hardware.**

LingBot-Map (Robbyant/lingbot-map) ships with two attention backends optimized for different deployment scenarios. Understanding when to activate the **FlashInfer backend** versus the PyTorch SDPA fallback is critical for optimizing throughput and preventing out-of-memory errors during long-sequence streaming reconstruction.

## What Is the FlashInfer Backend?

The FlashInfer backend is a paged KV-cache implementation that packs *patch* and *special* tokens into CUDA pages and executes the entire attention computation in a single JIT-compiled kernel. According to the source code in [`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py), this design splits the cache into three distinct streams:

- **Scale pages**: Always resident in memory
- **Sliding-window patch pages**: Rotating buffer for visual tokens
- **Append-only special stream**: Dedicated path for special tokens

This architecture prevents the O(N) memory growth typical of standard attention mechanisms, allowing the model to process 10,000+ frames without exhausting VRAM.

## When Should You Use the FlashInfer Backend?

### Production Streaming Inference

Use FlashInfer for all production-grade streaming reconstructions where latency matters. The kernel processes all Q-K-V interactions in a single CUDA call, eliminating Python-to-CUDA dispatch overhead. When compiled with `--compile`, the reference implementation achieves **~20 FPS** on 518×378 inputs, making it suitable for interactive demos and real-time applications.

### Long Sequence Processing

Choose FlashInfer when processing sequences exceeding thousands of frames. The paged KV cache design keeps memory bounded regardless of sequence length, unlike the SDPA backend which stores a full KV matrix for every frame.

### Installation Requirements

Activate FlashInfer only after confirming:

1. You have a CUDA-enabled NVIDIA GPU (compute capability SM 70 or newer recommended)
2. You installed the `flashinfer-python` wheel:

```bash
pip install --index-url https://pypi.org/simple flashinfer-python

```

3. For faster first-use JIT compilation, optionally install:

```bash
pip install flashinfer-jit-cache -f https://flashinfer.ai/whl/cu128/flashinfer-jit-cache/

```

## When to Use the SDPA Backend Instead

The SDPA (Scaled Dot Product Attention) backend serves as a fallback when FlashInfer is unavailable or unsuitable. Located in [`lingbot_map/layers/attention.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/attention.py), the SDPA path uses `torch.nn.functional.scaled_dot_product_attention` without paged caching.

### Hardware Limitations and Debugging

Switch to SDPA when:

- **GPU incompatibility**: Your device lacks the required compute capability (pre-Ada GPUs older than SM 70) or you are running on CPU-only hardware
- **JIT compilation failures**: FlashInfer's JIT fails in offline environments or restricted containers
- **Numerical verification**: You need FP32 precision checks by setting `force_fp32=True` in `FlashInferKVCacheManager`
- **Quick prototyping**: Running short test sequences where memory overhead is irrelevant

Explicitly force SDPA via the CLI flag:

```bash
python demo.py \
    --model_path /path/to/lingbot-map-long.pt \
    --image_folder example/courthouse \
    --use_sdpa

```

## Implementation Details From the Source Code

The backend selection logic is implemented across three key files:

**[`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py)**: Implements the `FlashInferKVCacheManager` class managing the two-stream paged cache (scale and patch/special tokens). This file handles the memory-efficient paging strategy that bounds GPU memory usage.

**[`lingbot_map/layers/attention.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/attention.py)**: Contains the runtime dispatch logic that checks `use_flashinfer` availability. If FlashInfer cannot be imported, the code raises a clear error and automatically switches to the SDPA path.

**[`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py)**: High-level streaming logic that sets `use_flashinfer = not use_sdpa` based on CLI arguments, coordinating the reconstruction pipeline with the selected backend.

**[`scripts/benchmark_gct_memory.py`](https://github.com/Robbyant/lingbot-map/blob/main/scripts/benchmark_gct_memory.py)**: Provides memory profiling utilities that quantify the VRAM savings of FlashInfer's paged approach versus standard SDPA allocation.

## How to Install and Configure FlashInfer

### Standard Installation and Usage

Install the required wheel and run the interactive demo with automatic backend detection:

```bash

# Install FlashInfer (default backend)

pip install --index-url https://pypi.org/simple flashinfer-python

# Run with FlashInfer automatically selected

python demo.py \
    --model_path /path/to/lingbot-map-long.pt \
    --image_folder example/courthouse \
    --mask_sky \
    --compile

```

### Benchmarking Both Backends

Compare performance characteristics using the provided profiling script:

```bash

# Benchmark FlashInfer

python gct_profile.py \
    --backend flashinfer \
    --dtype bf16 \
    --compile \
    --model_path /path/to/lingbot-map.pt

# Benchmark SDPA fallback

python gct_profile.py \
    --backend sdpa \
    --dtype bf16 \
    --model_path /path/to/lingbot-map.pt

```

The [`gct_profile.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_profile.py) script outputs per-step latency and peak GPU memory usage, making the speed-memory trade-off explicit.

## Summary

- **Use FlashInfer** for all production streaming inference on CUDA GPUs to achieve ~20 FPS and bounded memory usage via paged KV caching.
- **Install requirements** include `flashinfer-python` and a GPU with compute capability SM 70+.
- **Use SDPA** only as a fallback for incompatible hardware, debugging (FP32 precision), or CPU-only environments by passing `--use_sdpa`.
- **Key files** governing this behavior are [`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py) and [`lingbot_map/layers/attention.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/attention.py).
- **Benchmark** both backends using [`gct_profile.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_profile.py) to quantify latency and memory differences for your specific hardware.

## Frequently Asked Questions

### Can I use FlashInfer on CPU-only machines?

No. FlashInfer requires a CUDA-enabled NVIDIA GPU and the `flashinfer-python` package. On CPU-only machines or systems without compatible CUDA drivers, use the SDPA backend by passing `--use_sdpa` to force PyTorch native attention.

### How do I force LingBot-Map to use SDPA instead of FlashInfer?

Pass the `--use_sdpa` flag when running [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py) or set `use_sdpa=True` in the model configuration. The code in [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py) automatically sets `use_flashinfer = not use_sdpa`, routing attention computation to `torch.nn.functional.scaled_dot_product_attention`.

### What GPU compute capability is required for FlashInfer?

FlashInfer works best on GPUs with compute capability SM 70 or newer (Ada architecture and above). Older GPUs may fail during JIT compilation or show degraded performance, in which case you should fall back to SDPA.

### Does FlashInfer affect numerical precision compared to SDPA?

FlashInfer typically runs in BF16 for speed, while SDPA can be forced to FP32 for higher precision checks. If you need to debug numerical accuracy, set `force_fp32=True` in `FlashInferKVCacheManager` or switch to SDPA, which supports FP32 natively without additional configuration.