# SDPA vs FlashInfer Attention Backends in LingBot-Map: Architecture and Performance Comparison

> Compare SDPA and FlashInfer attention backends in LingBot-Map. Discover FlashInfer's constant-memory KV-cache vs SDPA's dense tensor approach for efficient model performance.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: performance
- Published: 2026-07-08

---

**FlashInfer provides a paged KV-cache implementation that enables constant-memory attention for long video streams, while SDPA uses a dense tensor approach with simpler dependencies but linear memory growth.**

The LingBot-Map repository implements two mutually exclusive attention computation backends for video stream processing: SDPA (Scaled Dot Product Attention) and FlashInfer. Understanding the architectural differences between these SDPA vs FlashInfer attention backends is essential for optimizing GPU memory usage and inference latency when deploying long-form video models.

## Implementation Architecture

### SDPA Dense Tensor Approach

In [`lingbot_map/layers/attention.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/attention.py), the `SDPAAttention` class (lines 595-618) implements attention using `torch.nn.functional.scaled_dot_product_attention`. This backend stores KV caches as dense tensors in a Python `dict`, concatenating cached keys and values with the current frame before invoking the fused CUDA kernel. When `fused_attn=False`, the implementation falls back to manual matrix multiplication rather than the fused kernel.

The eviction logic for long sequences is handled externally by `_apply_kv_cache_eviction`, which manages the dense cache dictionary but cannot change the fundamental linear memory scaling of the stored tensors.

### FlashInfer Paged KV-Cache System

The `FlashInferAttention` class leverages the third-party FlashInfer library, specifically calling `flashinfer.BatchPrefillWithPagedKVCacheWrapper`. The `FlashInferKVCacheManager` (defined in [`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py), lines 55-140) implements a two-tier paging architecture that separates recyclable patch pages from append-only special pages.

This design bounds memory consumption to **O(sliding_window)** complexity rather than growing with total sequence length, enabling the model to process video streams of arbitrary duration without exhausting GPU memory.

## Memory Layout and Scalability

SDPA stores the complete KV tensor for each frame with dimensions `tokens_per_frame × heads × head_dim`. As the video stream lengthens, this dense storage grows linearly, quickly becoming the bottleneck for long-form content.

FlashInfer's paged approach stores only visible pages—specifically scale frames, sliding window context, and special tokens. The manager automatically evicts old pages that fall outside the visible window, maintaining constant memory usage regardless of how long the video stream continues.

## Performance Characteristics

SDPA executes a single fused CUDA kernel on the entire dense KV matrix, delivering optimal performance for modest sequence lengths. However, it becomes memory-bound for very long videos where the KV cache exceeds GPU capacity.

FlashInfer's paged kernel can attend to millions of tokens with constant memory overhead, typically achieving **2-3× speedup** on long video streams compared to the dense approach. The paging overhead is amortized across the sequence length, making it superior for production video streaming workloads.

Precision modes differ between implementations. SDPA always operates in the model's native dtype (fp16, bf16, or fp32). FlashInfer defaults to fp16/bf16 but supports `force_fp32=True` mode, which gathers visible KV pages into dense tensors and falls back to standard SDPA computation for numerical consistency (see `FlashInferKVCacheManager.compute_attention`, lines 72-84).

## Configuration and Usage

Backend selection occurs at model instantiation via the `use_sdpa` flag. In [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py) (lines 229-324), the constructor determines the block class dynamically:

```python
self.use_flashinfer = not use_sdpa
GlobalBlockCls = SDPABlock if self.use_sdpa else FlashInferBlock

```

Command-line usage demonstrates the backend toggle:

```bash

# Default FlashInfer backend (paged KV-cache)

python demo.py --model gct_stream

# Force SDPA backend (dense tensors, no FlashInfer dependency)

python demo.py --model gct_stream --use-sdpa

```

Programmatic configuration:

```python
from lingbot_map.models.gct_stream import GCTStream

# Use SDPA for debugging or when FlashInfer is unavailable

model = GCTStream(use_sdpa=True)

# Use FlashInfer for long video streams (default behavior)

model = GCTStream(use_sdpa=False)

```

When FlashInfer is selected but the library is not installed, the constructor raises a clear `RuntimeError`. SDPA works out-of-the-box with standard PyTorch dependencies only.

## Summary

- **SDPA** uses dense tensor storage in a Python dictionary, providing simple, dependency-free attention for moderate sequence lengths and small-scale experiments.
- **FlashInfer** implements a sophisticated paged KV-cache system that maintains O(sliding_window) memory complexity, enabling scalable processing of long video streams without linear memory growth.
- **Backend selection** is controlled via the `use_sdpa` parameter in model constructors or the `--use-sdpa` command-line flag in [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py).
- **Precision control** differs between implementations: SDPA uses native model dtypes exclusively, while FlashInfer supports `force_fp32` fallback to SDPA for numerical consistency.
- **Performance trade-offs** favor SDPA for short sequences and FlashInfer for long-form video where dense KV caches would otherwise exhaust available GPU memory.

## Frequently Asked Questions

### When should I use SDPA over FlashInfer?

Use SDPA when running small-scale experiments, when FlashInfer is unavailable, or when debugging requires deterministic behavior without external dependencies. SDPA is also preferable for CPU inference or GPUs with limited memory where the overhead of paging management outweighs benefits for short sequences.

### Does FlashInfer require specific hardware?

FlashInfer requires NVIDIA GPUs and the FlashInfer library installation. If the library is missing, the `FlashInferAttention` constructor raises a `RuntimeError` during initialization. SDPA works on any hardware supporting PyTorch's attention functions, including CPUs and older GPU architectures.

### How does the precision mode differ between backends?

SDPA always operates in the model's native dtype (fp16, bf16, or fp32). FlashInfer defaults to fp16/bf16 but can force fp32 computation by gathering visible KV pages into dense tensors and invoking standard SDPA when `force_fp32=True` is configured in the `FlashInferKVCacheManager`, providing a precision-controlled reference implementation.

### Can I switch backends without modifying the source code?

Yes. The [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py) entry point exposes the `--use-sdpa` flag to toggle between backends at runtime. For programmatic use, instantiate models with `use_sdpa=True` to select SDPA or `use_sdpa=False` (default) for FlashInfer without changing the underlying source files in [`lingbot_map/layers/attention.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/attention.py).