# FlashInfer vs SDPA Attention Backends in LingBot-Map: Architecture and Performance Comparison

> Compare FlashInfer and SDPA attention backends in LingBot-Map. Discover FlashInfer's constant-memory for long streams and SDPA's dense-tensor approach for shorter sequences.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: performance
- Published: 2026-07-29

---

**SDPA provides a straightforward dense-tensor implementation suitable for short sequences and debugging, while FlashInfer enables constant-memory attention on long video streams through a sophisticated paged KV-cache that delivers 2-3× speedups on lengthy inputs.**

LingBot-Map implements dual attention backends to support diverse deployment scenarios, allowing developers to choose between **FlashInfer** and **SDPA** (Scaled Dot-Product Attention) based on memory constraints and sequence length requirements. Understanding the architectural differences between these backends is essential for optimizing inference performance on the Robbyant/lingbot-map repository.

## Core Architectural Differences

### Implementation Strategy

**SDPA** leverages `torch.nn.functional.scaled_dot_product_attention` (or manual matrix multiplication when `fused_attn=False`) and requires no external dependencies. In contrast, **FlashInfer** integrates the third-party FlashInfer library, specifically calling `flashinfer.BatchPrefillWithPagedKVCacheWrapper` as implemented in [`lingbot_map/layers/attention.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/attention.py).

### Memory Layout and Scalability

The SDPA backend stores the complete KV tensor for each frame with size scaling as `tokens_per_frame × heads × head_dim`, causing linear memory growth with sequence length. FlashInfer maintains only visible pages—including scale frames, sliding window, and special tokens—evicting old pages to maintain **O(sliding_window)** memory complexity regardless of stream duration.

## KV-Cache Management Approaches

### SDPA Dense Tensor Caching

In `SDPAAttention` (defined at lines 595-618 in [`lingbot_map/layers/attention.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/attention.py)), the KV cache operates as a simple Python `dict` storing dense tensors. During the forward pass (lines 614-618), cached K/V values are concatenated with the current frame and passed to the fused attention kernel. Eviction logic is handled externally by `_apply_kv_cache_eviction` before the attention computation.

### FlashInfer Paged Cache System

`FlashInferAttention` utilizes `FlashInferKVCacheManager` (lines 55-140 in [`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py)) to implement a **paged** cache architecture that separates recyclable *patch* pages from append-only *special* pages. This system allows the model to attend to millions of tokens while maintaining constant GPU memory usage through aggressive page eviction.

## Performance and Precision Characteristics

### Computational Throughput

SDPA executes a single fused CUDA kernel on dense matrices, delivering optimal performance for modest sequence lengths but becoming memory-bound on very long videos. FlashInfer's paged kernel architecture provides approximately **2-3× speedup** on long video streams by eliminating the memory bottleneck associated with growing KV caches.

### Numerical Precision and Fallbacks

SDPA operates strictly in the model's native dtype (fp16, bf16, or fp32 when `fused_attn=False`). FlashInfer defaults to fp16/bf16 but supports a `force_fp32=True` mode that gathers visible KV pages into dense tensors and falls back to standard SDPA (`compute_attention` lines 72-84 in [`flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/flashinfer_cache.py)), providing deterministic numerical precision when required.

## Backend Selection and Configuration

Backend selection occurs at model construction through the `use_sdpa` flag (see lines 229-324 in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py)). When `use_sdpa=False`, the architecture instantiates `FlashInferBlock`; otherwise, it uses `SDPABlock`.

```python

# Internal backend selection logic

self.use_flashinfer = not use_sdpa
GlobalBlockCls = SDPABlock if self.use_sdpa else FlashInferBlock

```

Command-line activation:

```bash

# Use FlashInfer (default for long video streams)

python demo.py --model gct_stream

# Force SDPA backend for debugging or CPU inference

python demo.py --model gct_stream --use-sdpa

```

Programmatic switching:

```python
from lingbot_map.models.gct_stream import GCTStream

# FlashInfer path (default)

model_flash = GCTStream(use_sdpa=False)

# SDPA path

model_sdpa = GCTStream(use_sdpa=True)

```

## Summary

- **SDPA** implements a dense-tensor attention mechanism using PyTorch's native `scaled_dot_product_attention`, storing KV caches as simple Python dictionaries with linear memory scaling.
- **FlashInfer** provides a paged KV-cache architecture via `FlashInferKVCacheManager`, enabling constant-memory attention on long sequences through the `BatchPrefillWithPagedKVCacheWrapper` kernel.
- SDPA requires no external dependencies and offers deterministic precision modes, making it suitable for debugging and small-scale experiments.
- FlashInfer delivers 2-3× performance improvements on lengthy video streams but requires the FlashInfer library installation and handles memory through sophisticated page eviction.
- Switch between backends using the `use_sdpa` boolean flag in model constructors or the `--use-sdpa` CLI argument in [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py).

## Frequently Asked Questions

### When should I use SDPA instead of FlashInfer?

Choose SDPA when running small-scale experiments, when FlashInfer is unavailable on your hardware, or when you require deterministic behavior without external dependencies. SDPA is also preferable for debugging attention mechanisms since it uses standard PyTorch operations in `SDPAAttention.forward` (lines 614-618) rather than optimized third-party kernels.

### Does FlashInfer support fp32 precision for numerical stability?

Yes. FlashInfer defaults to fp16/bf16, but you can force fp32 precision by setting `force_fp32=True` when constructing `FlashInferKVCacheManager`. In this mode, the manager gathers visible KV pages into dense tensors and executes `scaled_dot_product_attention` (lines 72-84 in [`flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/flashinfer_cache.py)), identical to the SDPA computation path.

### How does the paged cache in FlashInfer improve memory efficiency?

FlashInfer's `FlashInferKVCacheManager` (lines 55-140) stores only active pages visible within the attention window, evicting old pages to maintain **O(sliding_window)** memory complexity. This contrasts with SDPA's dense storage of all historical tokens, which grows linearly with sequence length and quickly exhausts GPU memory on long videos.

### What happens if FlashInfer is not installed when I try to use it?

The code raises a clear `RuntimeError` during the `FlashInferAttention` constructor if the FlashInfer library is unavailable. To avoid this, either install the FlashInfer dependency or explicitly select the SDPA backend using `--use-sdpa` when running [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py).