FlashInfer vs SDPA Backends for LingBot-Map Attention: Performance and Portability Trade-offs

FlashInfer delivers high-throughput paged KV-cache attention via custom CUDA kernels achieving 20+ FPS on long videos, while SDPA provides a portable PyTorch fallback requiring no extra dependencies but with lower performance on extended sequences.

LingBot-Map implements a streaming attention layer that supports two interchangeable compute backends. The repository Robbyant/lingbot-map allows users to choose between FlashInfer's optimized CUDA kernels and PyTorch's native SDPA implementation based on hardware constraints and real-time processing requirements.

Kernel Architecture and Throughput

The primary distinction between these backends lies in how they execute the attention computation at the CUDA kernel level.

FlashInfer Custom CUDA Kernels

The FlashInfer backend leverages the flashinfer package's BatchPrefillWithPagedKVCacheWrapper to execute attention. In lingbot_map/layers/attention.py, the FlashInferAttention class prepares query, key, and value tensors and delegates computation to the paged KV-cache manager. This approach yields significantly higher throughput, especially for long sequences with many frames.

On an RTX 4090 (SM90), the FlashInfer FA3 kernel achieves approximately 20 FPS for 518×378 resolution video at 10,000+ frames. Performance scales linearly with sequence length due to the efficient memory access patterns of the paged cache implementation.

SDPA Native PyTorch Implementation

The SDPA backend utilizes torch.nn.functional.scaled_dot_product_attention through the SDPAAttention class. This implementation runs on any CUDA-enabled GPU without requiring additional packages, making it ideal for rapid prototyping.

However, because the SDPA path materializes the entire KV cache each step and reshapes tensors during the forward pass, throughput degrades on very long sequences compared to FlashInfer. For streams under 300 frames, the performance difference is negligible, but extended sequences reveal the architectural limitations of the dict-based cache approach.

KV-Cache Management Strategies

Memory management represents the second major differentiator between these backends.

Paged Cache with FlashInferKVCacheManager

FlashInfer employs a sophisticated paging system via FlashInferKVCacheManager defined in lingbot_map/layers/flashinfer_cache.py. This manager stores key-value pairs in non-contiguous pages that can be appended, evicted, and read back without copying the entire cache each timestep.

The paging mechanism enables constant-time per-frame cost and minimizes memory fragmentation. When FlashInferAttention.forward processes a new frame, it appends to the existing cache pages and applies eviction policies without tensor reallocation.

Dict-Based Cache in SDPA

Conversely, SDPAAttention utilizes a plain Python dict of tensors to store cache states. Located in lingbot_map/layers/attention.py around lines 559-566, this implementation performs eviction by slicing and concatenating tensors, which requires materializing the full cache in GPU memory each step.

While functionally equivalent to the FlashInfer approach for correctness, the dict-based strategy incurs higher memory bandwidth usage and temporary allocations during the forward pass.

Installation Requirements and Fallback Behavior

Dependency management differs substantially between the two paths.

FlashInfer requires the optional flashinfer-python wheel, installable via pip install flashinfer-python. At import time in lingbot_map/layers/attention.py (lines 24-30), the code attempts to import flashinfer; if this fails, FLASHINFER_AVAILABLE is set to False and any instantiation of FlashInferAttention raises a clear runtime error.

SDPA requires no additional packages beyond PyTorch 2.8+ (or any version providing scaled_dot_product_attention). When FlashInfer is unavailable, the system automatically falls back to SDPA when explicitly requested via configuration flags.

Configuration and Usage Examples

The backend selection propagates through the model architecture via the use_sdpa boolean flag.

Python API Configuration

from lingbot_map.models.gct_stream import GCTStream

# FlashInfer backend (default, requires flashinfer package)

model_flash = GCTStream(
    img_size=518,
    patch_size=16,
    max_frame_num=10000,
    use_sdpa=False,
)

# SDPA backend (portable, no extra dependencies)

model_sdpa = GCTStream(
    img_size=518,
    patch_size=16,
    max_frame_num=10000,
    use_sdpa=True,
)

Command-Line Interface

# Default FlashInfer path

python demo.py --model_path lingbot-map-long.pt --image_folder example/loop

# Force SDPA fallback

python demo.py --use_sdpa --model_path lingbot-map-long.pt --image_folder example/loop

Direct Attention Class Instantiation

import torch
from lingbot_map.layers.attention import FlashInferAttention, SDPAAttention

x = torch.randn(1, 1024, 768).cuda()  # [B, N, C]

# FlashInfer path

flash_attn = FlashInferAttention(dim=768, num_heads=12)
out_flash = flash_attn(x)

# SDPA path

sdpa_attn = SDPAAttention(dim=768, num_heads=12)
out_sdpa = sdpa_attn(x)

Model Inspection

model = GCTStream(..., use_sdpa=args.use_sdpa)
print(type(model.aggregator.global_blocks[0]))

# Returns: <class 'lingbot_map.layers.block.FlashInferBlock'> when use_sdpa=False

# Returns: <class 'lingbot_map.layers.block.SDPABlock'> when use_sdpa=True

Implementation Entry Points

The backend decision flows through several key files in the repository:

1. Argument Parsing: demo.py (line 382) exposes the --use_sdpa flag, storing it in args.use_sdpa and passing it to the model constructor.

2. Model Construction: GCTStream in lingbot_map/models/gct_stream.py (lines 125-129) receives use_sdpa and forwards it to the AggregatorStream.

3. Aggregator Selection: In lingbot_map/aggregator/stream.py (lines 136-137), the initializer selects between FlashInferBlock and SDPABlock classes based on self.use_sdpa. This flag also determines whether streaming attention layers instantiate FlashInferAttention or SDPAAttention.

4. Forward Execution:

   • FlashInfer: FlashInferAttention.forward (lines 352-360) manages cache appending and eviction through the paged manager before calling manager.compute_attention.
   • SDPA: SDPAAttention.forward (lines 559-566) handles dict-based cache manipulation and invokes F.scaled_dot_product_attention.

Both implementations share identical public APIs and KV-cache eviction logic, ensuring functional parity while allowing performance-portability trade-offs.

Summary

• FlashInfer provides custom CUDA kernels with paged KV-cache management for maximum throughput on long video sequences, requiring the flashinfer-python package.
• SDPA offers a portable PyTorch implementation using scaled_dot_product_attention with dict-based caching, suitable for environments without FlashInfer support.
• Configuration flows through the use_sdpa flag, propagated from demo.py through GCTStream to AggregatorStream and finally to the attention layer classes.
• Both backends maintain API compatibility in FlashInferAttention and SDPAAttention, allowing seamless switching without model weight modifications.
• FlashInfer achieves approximately 20 FPS on RTX 4090 for high-resolution, long-duration videos, while SDPA exhibits linear performance degradation on sequences exceeding 300 frames.

Frequently Asked Questions

Which backend should I use for production workloads?

Use FlashInfer for production deployments requiring real-time processing of long video streams. The paged KV-cache implementation in FlashInferKVCacheManager provides constant-time per-frame processing and superior memory efficiency, delivering 20+ FPS on modern hardware (RTX 4090) even with 10,000+ frames.

Can I switch between backends without changing model weights?

Yes. The use_sdpa flag controls backend selection at runtime without requiring model retraining or checkpoint conversion. Both FlashInferAttention and SDPAAttention inherit from the same base Attention class and implement identical forward signatures, ensuring weight compatibility between FlashInferBlock and SDPABlock instantiations.

What happens if FlashInfer is not installed?

If the flashinfer package is unavailable, FLASHINFER_AVAILABLE is set to False during import in lingbot_map/layers/attention.py (lines 24-30). Attempting to instantiate FlashInferAttention raises a clear error, while the SDPA backend remains functional. You can force the SDPA path by setting use_sdpa=True when constructing GCTStream.

How does performance scale with sequence length?

FlashInfer scales linearly with sequence length due to its paged memory management, maintaining consistent frame rates even beyond 10,000 frames. SDPA performance degrades on longer sequences because SDPAAttention must reshape and copy the entire KV cache (stored as a Python dict) during each forward pass, making it suitable for shorter streams under 300 frames.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}