# FlashInfer vs SDPA Backends for LingBot-Map Attention: Performance and Portability Trade-offs

> Compare FlashInfer and SDPA backends for LingBot-Map attention. Discover performance and portability trade-offs, with FlashInfer achieving 20+ FPS on long videos and SDPA offering a portable PyTorch fallback.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: performance
- Published: 2026-07-06

---

**FlashInfer delivers high-throughput paged KV-cache attention via custom CUDA kernels achieving 20+ FPS on long videos, while SDPA provides a portable PyTorch fallback requiring no extra dependencies but with lower performance on extended sequences.**

LingBot-Map implements a streaming attention layer that supports two interchangeable compute backends. The repository Robbyant/lingbot-map allows users to choose between FlashInfer's optimized CUDA kernels and PyTorch's native SDPA implementation based on hardware constraints and real-time processing requirements.

## Kernel Architecture and Throughput

The primary distinction between these backends lies in how they execute the attention computation at the CUDA kernel level.

### FlashInfer Custom CUDA Kernels

The FlashInfer backend leverages the `flashinfer` package's `BatchPrefillWithPagedKVCacheWrapper` to execute attention. In [`lingbot_map/layers/attention.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/attention.py), the `FlashInferAttention` class prepares query, key, and value tensors and delegates computation to the paged KV-cache manager. This approach yields significantly higher throughput, especially for long sequences with many frames.

On an RTX 4090 (SM90), the FlashInfer FA3 kernel achieves approximately 20 FPS for 518×378 resolution video at 10,000+ frames. Performance scales linearly with sequence length due to the efficient memory access patterns of the paged cache implementation.

### SDPA Native PyTorch Implementation

The SDPA backend utilizes `torch.nn.functional.scaled_dot_product_attention` through the `SDPAAttention` class. This implementation runs on any CUDA-enabled GPU without requiring additional packages, making it ideal for rapid prototyping.

However, because the SDPA path materializes the entire KV cache each step and reshapes tensors during the forward pass, throughput degrades on very long sequences compared to FlashInfer. For streams under 300 frames, the performance difference is negligible, but extended sequences reveal the architectural limitations of the dict-based cache approach.

## KV-Cache Management Strategies

Memory management represents the second major differentiator between these backends.

### Paged Cache with FlashInferKVCacheManager

FlashInfer employs a sophisticated paging system via `FlashInferKVCacheManager` defined in [`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py). This manager stores key-value pairs in non-contiguous pages that can be appended, evicted, and read back without copying the entire cache each timestep.

The paging mechanism enables constant-time per-frame cost and minimizes memory fragmentation. When `FlashInferAttention.forward` processes a new frame, it appends to the existing cache pages and applies eviction policies without tensor reallocation.

### Dict-Based Cache in SDPA

Conversely, `SDPAAttention` utilizes a plain Python `dict` of tensors to store cache states. Located in [`lingbot_map/layers/attention.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/attention.py) around lines 559-566, this implementation performs eviction by slicing and concatenating tensors, which requires materializing the full cache in GPU memory each step.

While functionally equivalent to the FlashInfer approach for correctness, the dict-based strategy incurs higher memory bandwidth usage and temporary allocations during the forward pass.

## Installation Requirements and Fallback Behavior

Dependency management differs substantially between the two paths.

FlashInfer requires the optional `flashinfer-python` wheel, installable via `pip install flashinfer-python`. At import time in [`lingbot_map/layers/attention.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/attention.py) (lines 24-30), the code attempts to import `flashinfer`; if this fails, `FLASHINFER_AVAILABLE` is set to `False` and any instantiation of `FlashInferAttention` raises a clear runtime error.

SDPA requires no additional packages beyond PyTorch 2.8+ (or any version providing `scaled_dot_product_attention`). When FlashInfer is unavailable, the system automatically falls back to SDPA when explicitly requested via configuration flags.

## Configuration and Usage Examples

The backend selection propagates through the model architecture via the `use_sdpa` boolean flag.

### Python API Configuration

```python
from lingbot_map.models.gct_stream import GCTStream

# FlashInfer backend (default, requires flashinfer package)

model_flash = GCTStream(
    img_size=518,
    patch_size=16,
    max_frame_num=10000,
    use_sdpa=False,
)

# SDPA backend (portable, no extra dependencies)

model_sdpa = GCTStream(
    img_size=518,
    patch_size=16,
    max_frame_num=10000,
    use_sdpa=True,
)

```

### Command-Line Interface

```bash

# Default FlashInfer path

python demo.py --model_path lingbot-map-long.pt --image_folder example/loop

# Force SDPA fallback

python demo.py --use_sdpa --model_path lingbot-map-long.pt --image_folder example/loop

```

### Direct Attention Class Instantiation

```python
import torch
from lingbot_map.layers.attention import FlashInferAttention, SDPAAttention

x = torch.randn(1, 1024, 768).cuda()  # [B, N, C]

# FlashInfer path

flash_attn = FlashInferAttention(dim=768, num_heads=12)
out_flash = flash_attn(x)

# SDPA path

sdpa_attn = SDPAAttention(dim=768, num_heads=12)
out_sdpa = sdpa_attn(x)

```

### Model Inspection

```python
model = GCTStream(..., use_sdpa=args.use_sdpa)
print(type(model.aggregator.global_blocks[0]))

# Returns: <class 'lingbot_map.layers.block.FlashInferBlock'> when use_sdpa=False

# Returns: <class 'lingbot_map.layers.block.SDPABlock'> when use_sdpa=True

```

## Implementation Entry Points

The backend decision flows through several key files in the repository:

1. **Argument Parsing**: [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py) (line 382) exposes the `--use_sdpa` flag, storing it in `args.use_sdpa` and passing it to the model constructor.

2. **Model Construction**: `GCTStream` in [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py) (lines 125-129) receives `use_sdpa` and forwards it to the `AggregatorStream`.

3. **Aggregator Selection**: In [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py) (lines 136-137), the initializer selects between `FlashInferBlock` and `SDPABlock` classes based on `self.use_sdpa`. This flag also determines whether streaming attention layers instantiate `FlashInferAttention` or `SDPAAttention`.

4. **Forward Execution**: 
   - **FlashInfer**: `FlashInferAttention.forward` (lines 352-360) manages cache appending and eviction through the paged manager before calling `manager.compute_attention`.
   - **SDPA**: `SDPAAttention.forward` (lines 559-566) handles dict-based cache manipulation and invokes `F.scaled_dot_product_attention`.

Both implementations share identical public APIs and KV-cache eviction logic, ensuring functional parity while allowing performance-portability trade-offs.

## Summary

- **FlashInfer** provides custom CUDA kernels with paged KV-cache management for maximum throughput on long video sequences, requiring the `flashinfer-python` package.
- **SDPA** offers a portable PyTorch implementation using `scaled_dot_product_attention` with dict-based caching, suitable for environments without FlashInfer support.
- Configuration flows through the `use_sdpa` flag, propagated from [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py) through `GCTStream` to `AggregatorStream` and finally to the attention layer classes.
- Both backends maintain API compatibility in `FlashInferAttention` and `SDPAAttention`, allowing seamless switching without model weight modifications.
- FlashInfer achieves approximately 20 FPS on RTX 4090 for high-resolution, long-duration videos, while SDPA exhibits linear performance degradation on sequences exceeding 300 frames.

## Frequently Asked Questions

### Which backend should I use for production workloads?

Use **FlashInfer** for production deployments requiring real-time processing of long video streams. The paged KV-cache implementation in `FlashInferKVCacheManager` provides constant-time per-frame processing and superior memory efficiency, delivering 20+ FPS on modern hardware (RTX 4090) even with 10,000+ frames.

### Can I switch between backends without changing model weights?

Yes. The `use_sdpa` flag controls backend selection at runtime without requiring model retraining or checkpoint conversion. Both `FlashInferAttention` and `SDPAAttention` inherit from the same base `Attention` class and implement identical forward signatures, ensuring weight compatibility between `FlashInferBlock` and `SDPABlock` instantiations.

### What happens if FlashInfer is not installed?

If the `flashinfer` package is unavailable, `FLASHINFER_AVAILABLE` is set to `False` during import in [`lingbot_map/layers/attention.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/attention.py) (lines 24-30). Attempting to instantiate `FlashInferAttention` raises a clear error, while the SDPA backend remains functional. You can force the SDPA path by setting `use_sdpa=True` when constructing `GCTStream`.

### How does performance scale with sequence length?

FlashInfer scales linearly with sequence length due to its paged memory management, maintaining consistent frame rates even beyond 10,000 frames. SDPA performance degrades on longer sequences because `SDPAAttention` must reshape and copy the entire KV cache (stored as a Python dict) during each forward pass, making it suitable for shorter streams under 300 frames.