LingBot-Map SDPA vs FlashInfer Backends: Performance and Memory Analysis

FlashInfer delivers approximately 5 FPS higher throughput and significantly lower GPU memory usage than SDPA by utilizing a paged KV-cache and optimized FA2/FA3 kernels, while SDPA provides a portable fallback using PyTorch's native scaled_dot_product_attention.

LingBot-Map supports two attention backends for streaming transformer inference: PyTorch's Scaled-Dot-Product Attention (SDPA) and the specialized FlashInfer library. Understanding the performance differences between LingBot-Map SDPA and FlashInfer backends is essential for optimizing production pipelines versus debugging environments. This analysis examines the architectural distinctions, throughput benchmarks, and memory characteristics based on the actual implementation in the Robbyant/lingbot-map repository.

Implementation Architecture

The backend selection determines how the KV-cache is managed and which attention kernels execute during streaming inference.

SDPA (PyTorch Native)

The SDPA backend uses torch.nn.functional.scaled_dot_product_attention operating on dense tensors assembled from a Python dictionary KV-cache. In lingbot_map/aggregator/stream.py (line 182), the model initializes self.kv_cache as a simple dictionary that updates frame-by-frame. During each forward pass, the _gather_kv method collects keys and values into dense tensors before passing them to PyTorch's generic attention function. This approach requires no external dependencies beyond PyTorch ≥ 2.0 and runs on any CUDA-capable GPU.

FlashInfer (Paged KV-Cache)

FlashInfer employs a custom FlashInferKVCacheManager defined in lingbot_map/layers/flashinfer_cache.py that stores keys and values in a page-aligned CUDA buffer. The implementation writes patches directly into pre-allocated memory pages (see lines 998–1005 in the cache manager), avoiding repeated Python-to-CUDA dispatches. The attention computation uses FlashInfer's FA2/FA3 kernels via BatchPrefillWithPagedKVCacheWrapper, invoked in lingbot_map/layers/attention.py through the prefill_wrapper.run method. This specialized path requires the external flashinfer package installed via pip install flashinfer.

Performance Benchmarks

Throughput differs significantly between the two backends at typical input resolutions.

FlashInfer achieves approximately 5 FPS higher frame rates than SDPA when processing 518×378 resolution video streams, according to the performance notes in demo.py. This acceleration stems from the kernel's block-wise, mixed-precision execution that bypasses PyTorch's generic dispatch overhead.

SDPA incurs additional latency due to the gather-and-attention overhead: each frame requires assembling the full KV tensors from the dictionary cache before computing attention. While functional, this dense tensor approach cannot match the optimized memory access patterns of FlashInfer's paged architecture.

Memory Usage Characteristics

Memory efficiency represents the most critical operational difference between the backends.

SDPA allocates dense tensors for each frame's computation. The dictionary-based cache can hold the full history, but memory grows with the number of tokens per frame and the sliding window length. This results in higher peak GPU memory consumption, as the system must gather and maintain contiguous tensors for the attention operation.

FlashInfer reduces memory pressure dramatically through its paged cache manager. Only the pages actually needed for the current sliding window remain allocated; the system recycles pages automatically based on window parameters. The scripts/benchmark_gct_memory.py script provides quantitative measurements showing substantially lower peak memory usage compared to the SDPA path, making it suitable for long-sequence streaming applications.

Precision and Hardware Requirements

Numerical precision support varies between backends and affects compatibility.

SDPA supports both fp32 and bf16 computations natively, depending on the model's dtype configuration. This flexibility makes it suitable for debugging scenarios requiring full precision or environments where reduced precision is unacceptable.

FlashInfer FA2 kernels support fp16/bf16 only. When the model runs in fp32 mode, the FlashInferKVCacheManager automatically switches to a gather-plus-SDPA path (triggered when force_fp32=True), effectively falling back to PyTorch's implementation for unsupported precision. Hardware requirements also restrict FlashInfer to newer GPUs: SM 80+ for FA3 kernels or SM 70+ for FA2, whereas SDPA works on any CUDA GPU.

How to Select the Backend

Choose the backend during model initialization or via command-line flags.

from lingbot_map import LingBotMap

# Use FlashInfer (default, recommended for production)

model = LingBotMap(
    img_size=518,
    patch_size=16,
    embed_dim=768,
    use_sdpa=False,
)

# Force SDPA (portable fallback)

model_sdpa = LingBotMap(
    img_size=518,
    patch_size=16,
    embed_dim=768,
    use_sdpa=True,
)

# Runtime inference uses identical API for both

frames = torch.randn(1, 10, 3, 518, 518).cuda()
outputs_flash = model(frames)
outputs_sdpa = model_sdpa(frames)

Command-line selection via demo.py:

# FlashInfer (default)

python demo.py --model-path checkpoint.pth

# Force SDPA

python demo.py --model-path checkpoint.pth --sdpa

Inspect the active cache manager at runtime:

if hasattr(model.aggregator, "kv_cache_manager"):
    print("FlashInfer KV cache in use")
else:
    print("Using dict-based SDPA cache")

Summary

• FlashInfer provides superior performance with approximately 5 FPS higher throughput and dramatically reduced memory usage through its paged KV-cache and FA2/FA3 kernels, making it ideal for production streaming inference.
• SDPA offers a dependency-free fallback that works on any CUDA GPU with PyTorch ≥ 2.0, supporting both fp32 and bf16 precision, but incurs higher memory overhead and lower frame rates.
• Backend selection occurs via the use_sdpa flag in lingbot_map/models/gct_stream.py or the --sdpa command-line argument in demo.py.
• Precision constraints force FlashInfer to fall back to SDPA when running in fp32 mode, as FA2 kernels only support fp16/bf16.

Frequently Asked Questions

When should I use SDPA instead of FlashInfer?

Use SDPA when FlashInfer installation is impossible due to environment constraints, when debugging requires pure PyTorch code without custom kernels, or when running on older GPUs below SM 70 that lack FlashInfer support. SDPA also serves as the automatic fallback when requesting fp32 precision, as FlashInfer's FA2 kernels only support fp16 and bf16.

Does FlashInfer require specific GPU hardware?

Yes. FlashInfer FA2 requires SM 70+ (Volta architecture or newer), while FA3 requires SM 80+ (Ampere architecture or newer). SDPA runs on any CUDA-capable GPU supported by PyTorch ≥ 2.0, making it more portable across older hardware generations.

How much memory does FlashInfer save compared to SDPA?

FlashInfer's paged KV-cache manager stores only active pages in a pre-allocated CUDA buffer, recycling memory automatically based on sliding-window parameters. This architecture reduces peak GPU memory significantly compared to SDPA's dense tensor gathering, though exact savings depend on sequence length and batch size. Run scripts/benchmark_gct_memory.py to measure specific memory reductions for your configuration.

Why is FlashInfer faster than SDPA?

FlashInfer achieves higher throughput through three optimizations: (1) a paged KV-cache manager that eliminates Python-to-CUDA dispatch overhead for each frame, (2) specialized FA2/FA3 kernels that execute attention in a single launch with block-wise memory access patterns, and (3) reduced memory traffic by attending only to visible pages rather than gathering full dense tensors. These optimizations yield approximately 5 FPS higher performance at 518×378 resolution compared to PyTorch's generic SDPA implementation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}