# LingBot-Map SDPA vs FlashInfer Backends: Performance and Memory Analysis

> Compare LingBot Map's SDPA vs FlashInfer backends. Discover FlashInfer's higher FPS and lower memory usage thanks to optimized kernels and paged KV-cache. Learn more.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: performance
- Published: 2026-06-29

---

**FlashInfer delivers approximately 5 FPS higher throughput and significantly lower GPU memory usage than SDPA by utilizing a paged KV-cache and optimized FA2/FA3 kernels, while SDPA provides a portable fallback using PyTorch's native `scaled_dot_product_attention`.**

LingBot-Map supports two attention backends for streaming transformer inference: PyTorch's Scaled-Dot-Product Attention (SDPA) and the specialized FlashInfer library. Understanding the performance differences between LingBot-Map SDPA and FlashInfer backends is essential for optimizing production pipelines versus debugging environments. This analysis examines the architectural distinctions, throughput benchmarks, and memory characteristics based on the actual implementation in the `Robbyant/lingbot-map` repository.

## Implementation Architecture

The backend selection determines how the KV-cache is managed and which attention kernels execute during streaming inference.

### SDPA (PyTorch Native)

The SDPA backend uses `torch.nn.functional.scaled_dot_product_attention` operating on dense tensors assembled from a Python dictionary KV-cache. In [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py) (line 182), the model initializes `self.kv_cache` as a simple dictionary that updates frame-by-frame. During each forward pass, the `_gather_kv` method collects keys and values into dense tensors before passing them to PyTorch's generic attention function. This approach requires no external dependencies beyond PyTorch ≥ 2.0 and runs on any CUDA-capable GPU.

### FlashInfer (Paged KV-Cache)

FlashInfer employs a custom `FlashInferKVCacheManager` defined in [`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py) that stores keys and values in a page-aligned CUDA buffer. The implementation writes patches directly into pre-allocated memory pages (see lines 998–1005 in the cache manager), avoiding repeated Python-to-CUDA dispatches. The attention computation uses FlashInfer's FA2/FA3 kernels via `BatchPrefillWithPagedKVCacheWrapper`, invoked in [`lingbot_map/layers/attention.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/attention.py) through the `prefill_wrapper.run` method. This specialized path requires the external `flashinfer` package installed via `pip install flashinfer`.

## Performance Benchmarks

Throughput differs significantly between the two backends at typical input resolutions.

**FlashInfer** achieves approximately **5 FPS higher** frame rates than SDPA when processing 518×378 resolution video streams, according to the performance notes in [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py). This acceleration stems from the kernel's block-wise, mixed-precision execution that bypasses PyTorch's generic dispatch overhead.

**SDPA** incurs additional latency due to the gather-and-attention overhead: each frame requires assembling the full KV tensors from the dictionary cache before computing attention. While functional, this dense tensor approach cannot match the optimized memory access patterns of FlashInfer's paged architecture.

## Memory Usage Characteristics

Memory efficiency represents the most critical operational difference between the backends.

**SDPA** allocates dense tensors for each frame's computation. The dictionary-based cache can hold the full history, but memory grows with the number of tokens per frame and the sliding window length. This results in higher peak GPU memory consumption, as the system must gather and maintain contiguous tensors for the attention operation.

**FlashInfer** reduces memory pressure dramatically through its paged cache manager. Only the pages actually needed for the current sliding window remain allocated; the system recycles pages automatically based on window parameters. The [`scripts/benchmark_gct_memory.py`](https://github.com/Robbyant/lingbot-map/blob/main/scripts/benchmark_gct_memory.py) script provides quantitative measurements showing substantially lower peak memory usage compared to the SDPA path, making it suitable for long-sequence streaming applications.

## Precision and Hardware Requirements

Numerical precision support varies between backends and affects compatibility.

**SDPA** supports both **fp32** and **bf16** computations natively, depending on the model's dtype configuration. This flexibility makes it suitable for debugging scenarios requiring full precision or environments where reduced precision is unacceptable.

**FlashInfer** FA2 kernels support **fp16/bf16** only. When the model runs in fp32 mode, the `FlashInferKVCacheManager` automatically switches to a gather-plus-SDPA path (triggered when `force_fp32=True`), effectively falling back to PyTorch's implementation for unsupported precision. Hardware requirements also restrict FlashInfer to newer GPUs: SM 80+ for FA3 kernels or SM 70+ for FA2, whereas SDPA works on any CUDA GPU.

## How to Select the Backend

Choose the backend during model initialization or via command-line flags.

```python
from lingbot_map import LingBotMap

# Use FlashInfer (default, recommended for production)

model = LingBotMap(
    img_size=518,
    patch_size=16,
    embed_dim=768,
    use_sdpa=False,
)

# Force SDPA (portable fallback)

model_sdpa = LingBotMap(
    img_size=518,
    patch_size=16,
    embed_dim=768,
    use_sdpa=True,
)

# Runtime inference uses identical API for both

frames = torch.randn(1, 10, 3, 518, 518).cuda()
outputs_flash = model(frames)
outputs_sdpa = model_sdpa(frames)

```

Command-line selection via [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py):

```bash

# FlashInfer (default)

python demo.py --model-path checkpoint.pth

# Force SDPA

python demo.py --model-path checkpoint.pth --sdpa

```

Inspect the active cache manager at runtime:

```python
if hasattr(model.aggregator, "kv_cache_manager"):
    print("FlashInfer KV cache in use")
else:
    print("Using dict-based SDPA cache")

```

## Summary

- **FlashInfer** provides superior performance with approximately 5 FPS higher throughput and dramatically reduced memory usage through its paged KV-cache and FA2/FA3 kernels, making it ideal for production streaming inference.
- **SDPA** offers a dependency-free fallback that works on any CUDA GPU with PyTorch ≥ 2.0, supporting both fp32 and bf16 precision, but incurs higher memory overhead and lower frame rates.
- **Backend selection** occurs via the `use_sdpa` flag in [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py) or the `--sdpa` command-line argument in [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py).
- **Precision constraints** force FlashInfer to fall back to SDPA when running in fp32 mode, as FA2 kernels only support fp16/bf16.

## Frequently Asked Questions

### When should I use SDPA instead of FlashInfer?

Use SDPA when FlashInfer installation is impossible due to environment constraints, when debugging requires pure PyTorch code without custom kernels, or when running on older GPUs below SM 70 that lack FlashInfer support. SDPA also serves as the automatic fallback when requesting fp32 precision, as FlashInfer's FA2 kernels only support fp16 and bf16.

### Does FlashInfer require specific GPU hardware?

Yes. FlashInfer FA2 requires SM 70+ (Volta architecture or newer), while FA3 requires SM 80+ (Ampere architecture or newer). SDPA runs on any CUDA-capable GPU supported by PyTorch ≥ 2.0, making it more portable across older hardware generations.

### How much memory does FlashInfer save compared to SDPA?

FlashInfer's paged KV-cache manager stores only active pages in a pre-allocated CUDA buffer, recycling memory automatically based on sliding-window parameters. This architecture reduces peak GPU memory significantly compared to SDPA's dense tensor gathering, though exact savings depend on sequence length and batch size. Run [`scripts/benchmark_gct_memory.py`](https://github.com/Robbyant/lingbot-map/blob/main/scripts/benchmark_gct_memory.py) to measure specific memory reductions for your configuration.

### Why is FlashInfer faster than SDPA?

FlashInfer achieves higher throughput through three optimizations: (1) a paged KV-cache manager that eliminates Python-to-CUDA dispatch overhead for each frame, (2) specialized FA2/FA3 kernels that execute attention in a single launch with block-wise memory access patterns, and (3) reduced memory traffic by attending only to visible pages rather than gathering full dense tensors. These optimizations yield approximately 5 FPS higher performance at 518×378 resolution compared to PyTorch's generic SDPA implementation.