FlashInfer vs SDPA Attention Backends in LingBot-Map: Architecture and Performance Comparison

SDPA provides a straightforward dense-tensor implementation suitable for short sequences and debugging, while FlashInfer enables constant-memory attention on long video streams through a sophisticated paged KV-cache that delivers 2-3× speedups on lengthy inputs.

LingBot-Map implements dual attention backends to support diverse deployment scenarios, allowing developers to choose between FlashInfer and SDPA (Scaled Dot-Product Attention) based on memory constraints and sequence length requirements. Understanding the architectural differences between these backends is essential for optimizing inference performance on the Robbyant/lingbot-map repository.

Core Architectural Differences

Implementation Strategy

SDPA leverages torch.nn.functional.scaled_dot_product_attention (or manual matrix multiplication when fused_attn=False) and requires no external dependencies. In contrast, FlashInfer integrates the third-party FlashInfer library, specifically calling flashinfer.BatchPrefillWithPagedKVCacheWrapper as implemented in lingbot_map/layers/attention.py.

Memory Layout and Scalability

The SDPA backend stores the complete KV tensor for each frame with size scaling as tokens_per_frame × heads × head_dim, causing linear memory growth with sequence length. FlashInfer maintains only visible pages—including scale frames, sliding window, and special tokens—evicting old pages to maintain O(sliding_window) memory complexity regardless of stream duration.

KV-Cache Management Approaches

SDPA Dense Tensor Caching

In SDPAAttention (defined at lines 595-618 in lingbot_map/layers/attention.py), the KV cache operates as a simple Python dict storing dense tensors. During the forward pass (lines 614-618), cached K/V values are concatenated with the current frame and passed to the fused attention kernel. Eviction logic is handled externally by _apply_kv_cache_eviction before the attention computation.

FlashInfer Paged Cache System

FlashInferAttention utilizes FlashInferKVCacheManager (lines 55-140 in lingbot_map/layers/flashinfer_cache.py) to implement a paged cache architecture that separates recyclable patch pages from append-only special pages. This system allows the model to attend to millions of tokens while maintaining constant GPU memory usage through aggressive page eviction.

Performance and Precision Characteristics

Computational Throughput

SDPA executes a single fused CUDA kernel on dense matrices, delivering optimal performance for modest sequence lengths but becoming memory-bound on very long videos. FlashInfer's paged kernel architecture provides approximately 2-3× speedup on long video streams by eliminating the memory bottleneck associated with growing KV caches.

Numerical Precision and Fallbacks

SDPA operates strictly in the model's native dtype (fp16, bf16, or fp32 when fused_attn=False). FlashInfer defaults to fp16/bf16 but supports a force_fp32=True mode that gathers visible KV pages into dense tensors and falls back to standard SDPA (compute_attention lines 72-84 in flashinfer_cache.py), providing deterministic numerical precision when required.

Backend Selection and Configuration

Backend selection occurs at model construction through the use_sdpa flag (see lines 229-324 in lingbot_map/models/gct_stream_window_v2.py). When use_sdpa=False, the architecture instantiates FlashInferBlock; otherwise, it uses SDPABlock.

# Internal backend selection logic

self.use_flashinfer = not use_sdpa
GlobalBlockCls = SDPABlock if self.use_sdpa else FlashInferBlock

Command-line activation:

# Use FlashInfer (default for long video streams)

python demo.py --model gct_stream

# Force SDPA backend for debugging or CPU inference

python demo.py --model gct_stream --use-sdpa

Programmatic switching:

from lingbot_map.models.gct_stream import GCTStream

# FlashInfer path (default)

model_flash = GCTStream(use_sdpa=False)

# SDPA path

model_sdpa = GCTStream(use_sdpa=True)

Summary

• SDPA implements a dense-tensor attention mechanism using PyTorch's native scaled_dot_product_attention, storing KV caches as simple Python dictionaries with linear memory scaling.
• FlashInfer provides a paged KV-cache architecture via FlashInferKVCacheManager, enabling constant-memory attention on long sequences through the BatchPrefillWithPagedKVCacheWrapper kernel.
• SDPA requires no external dependencies and offers deterministic precision modes, making it suitable for debugging and small-scale experiments.
• FlashInfer delivers 2-3× performance improvements on lengthy video streams but requires the FlashInfer library installation and handles memory through sophisticated page eviction.
• Switch between backends using the use_sdpa boolean flag in model constructors or the --use-sdpa CLI argument in demo.py.

Frequently Asked Questions

When should I use SDPA instead of FlashInfer?

Choose SDPA when running small-scale experiments, when FlashInfer is unavailable on your hardware, or when you require deterministic behavior without external dependencies. SDPA is also preferable for debugging attention mechanisms since it uses standard PyTorch operations in SDPAAttention.forward (lines 614-618) rather than optimized third-party kernels.

Does FlashInfer support fp32 precision for numerical stability?

Yes. FlashInfer defaults to fp16/bf16, but you can force fp32 precision by setting force_fp32=True when constructing FlashInferKVCacheManager. In this mode, the manager gathers visible KV pages into dense tensors and executes scaled_dot_product_attention (lines 72-84 in flashinfer_cache.py), identical to the SDPA computation path.

How does the paged cache in FlashInfer improve memory efficiency?

FlashInfer's FlashInferKVCacheManager (lines 55-140) stores only active pages visible within the attention window, evicting old pages to maintain O(sliding_window) memory complexity. This contrasts with SDPA's dense storage of all historical tokens, which grows linearly with sequence length and quickly exhausts GPU memory on long videos.

What happens if FlashInfer is not installed when I try to use it?

The code raises a clear RuntimeError during the FlashInferAttention constructor if the FlashInfer library is unavailable. To avoid this, either install the FlashInfer dependency or explicitly select the SDPA backend using --use-sdpa when running demo.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}