SDPA vs FlashInfer Attention Backends in LingBot-Map: Architecture and Performance Comparison

FlashInfer provides a paged KV-cache implementation that enables constant-memory attention for long video streams, while SDPA uses a dense tensor approach with simpler dependencies but linear memory growth.

The LingBot-Map repository implements two mutually exclusive attention computation backends for video stream processing: SDPA (Scaled Dot Product Attention) and FlashInfer. Understanding the architectural differences between these SDPA vs FlashInfer attention backends is essential for optimizing GPU memory usage and inference latency when deploying long-form video models.

Implementation Architecture

SDPA Dense Tensor Approach

In lingbot_map/layers/attention.py, the SDPAAttention class (lines 595-618) implements attention using torch.nn.functional.scaled_dot_product_attention. This backend stores KV caches as dense tensors in a Python dict, concatenating cached keys and values with the current frame before invoking the fused CUDA kernel. When fused_attn=False, the implementation falls back to manual matrix multiplication rather than the fused kernel.

The eviction logic for long sequences is handled externally by _apply_kv_cache_eviction, which manages the dense cache dictionary but cannot change the fundamental linear memory scaling of the stored tensors.

FlashInfer Paged KV-Cache System

The FlashInferAttention class leverages the third-party FlashInfer library, specifically calling flashinfer.BatchPrefillWithPagedKVCacheWrapper. The FlashInferKVCacheManager (defined in lingbot_map/layers/flashinfer_cache.py, lines 55-140) implements a two-tier paging architecture that separates recyclable patch pages from append-only special pages.

This design bounds memory consumption to O(sliding_window) complexity rather than growing with total sequence length, enabling the model to process video streams of arbitrary duration without exhausting GPU memory.

Memory Layout and Scalability

SDPA stores the complete KV tensor for each frame with dimensions tokens_per_frame × heads × head_dim. As the video stream lengthens, this dense storage grows linearly, quickly becoming the bottleneck for long-form content.

FlashInfer's paged approach stores only visible pages—specifically scale frames, sliding window context, and special tokens. The manager automatically evicts old pages that fall outside the visible window, maintaining constant memory usage regardless of how long the video stream continues.

Performance Characteristics

SDPA executes a single fused CUDA kernel on the entire dense KV matrix, delivering optimal performance for modest sequence lengths. However, it becomes memory-bound for very long videos where the KV cache exceeds GPU capacity.

FlashInfer's paged kernel can attend to millions of tokens with constant memory overhead, typically achieving 2-3× speedup on long video streams compared to the dense approach. The paging overhead is amortized across the sequence length, making it superior for production video streaming workloads.

Precision modes differ between implementations. SDPA always operates in the model's native dtype (fp16, bf16, or fp32). FlashInfer defaults to fp16/bf16 but supports force_fp32=True mode, which gathers visible KV pages into dense tensors and falls back to standard SDPA computation for numerical consistency (see FlashInferKVCacheManager.compute_attention, lines 72-84).

Configuration and Usage

Backend selection occurs at model instantiation via the use_sdpa flag. In lingbot_map/models/gct_stream_window_v2.py (lines 229-324), the constructor determines the block class dynamically:

self.use_flashinfer = not use_sdpa
GlobalBlockCls = SDPABlock if self.use_sdpa else FlashInferBlock

Command-line usage demonstrates the backend toggle:

# Default FlashInfer backend (paged KV-cache)

python demo.py --model gct_stream

# Force SDPA backend (dense tensors, no FlashInfer dependency)

python demo.py --model gct_stream --use-sdpa

Programmatic configuration:

from lingbot_map.models.gct_stream import GCTStream

# Use SDPA for debugging or when FlashInfer is unavailable

model = GCTStream(use_sdpa=True)

# Use FlashInfer for long video streams (default behavior)

model = GCTStream(use_sdpa=False)

When FlashInfer is selected but the library is not installed, the constructor raises a clear RuntimeError. SDPA works out-of-the-box with standard PyTorch dependencies only.

Summary

• SDPA uses dense tensor storage in a Python dictionary, providing simple, dependency-free attention for moderate sequence lengths and small-scale experiments.
• FlashInfer implements a sophisticated paged KV-cache system that maintains O(sliding_window) memory complexity, enabling scalable processing of long video streams without linear memory growth.
• Backend selection is controlled via the use_sdpa parameter in model constructors or the --use-sdpa command-line flag in demo.py.
• Precision control differs between implementations: SDPA uses native model dtypes exclusively, while FlashInfer supports force_fp32 fallback to SDPA for numerical consistency.
• Performance trade-offs favor SDPA for short sequences and FlashInfer for long-form video where dense KV caches would otherwise exhaust available GPU memory.

Frequently Asked Questions

When should I use SDPA over FlashInfer?

Use SDPA when running small-scale experiments, when FlashInfer is unavailable, or when debugging requires deterministic behavior without external dependencies. SDPA is also preferable for CPU inference or GPUs with limited memory where the overhead of paging management outweighs benefits for short sequences.

Does FlashInfer require specific hardware?

FlashInfer requires NVIDIA GPUs and the FlashInfer library installation. If the library is missing, the FlashInferAttention constructor raises a RuntimeError during initialization. SDPA works on any hardware supporting PyTorch's attention functions, including CPUs and older GPU architectures.

How does the precision mode differ between backends?

SDPA always operates in the model's native dtype (fp16, bf16, or fp32). FlashInfer defaults to fp16/bf16 but can force fp32 computation by gathering visible KV pages into dense tensors and invoking standard SDPA when force_fp32=True is configured in the FlashInferKVCacheManager, providing a precision-controlled reference implementation.

Can I switch backends without modifying the source code?

Yes. The demo.py entry point exposes the --use-sdpa flag to toggle between backends at runtime. For programmatic use, instantiate models with use_sdpa=True to select SDPA or use_sdpa=False (default) for FlashInfer without changing the underlying source files in lingbot_map/layers/attention.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}