How to Force LingBot-Map to Use the SDPA Backend Instead of FlashInfer

Set use_sdpa=True when instantiating the model or pass the --use_sdpa command-line flag to switch from FlashInfer to PyTorch's native scaled-dot-product attention implementation.

LingBot-Map (from the Robbyant/lingbot-map repository) provides two distinct attention backends for streaming inference: the default FlashInfer library with paged KV-cache support, and the SDPA (Scaled-Dot-Product Attention) backend using native PyTorch operations. This guide details exactly how to force LingBot-Map to use the SDPA backend instead of FlashInfer, removing the FlashInfer dependency while preserving inference capabilities.

Understanding the Backend Architecture

LingBot-Map's streaming aggregator selects attention implementations based on the use_sdpa boolean flag. By default, use_sdpa=False enables FlashInferBlock and the FlashInferKVCacheManager for optimized memory management. When you force SDPA mode, the system instantiates SDPABlock classes and utilizes a simple Python dictionary-based KV cache instead.

The backend selection logic resides in lingbot_map/aggregator/stream.py, where the constructor processes the use_sdpa keyword argument (lines 88-94):

use_sdpa = kwargs.pop('use_sdpa', False)       # default value

self.use_sdpa = use_sdpa
self.use_flashinfer = not use_sdpa              # FlashInfer disabled when use_sdpa=True

This flag propagates through lingbot_map/models/gct_stream.py (lines 220-225), which forwards the configuration to the aggregator during _build_aggregator.

Method 1: Force SDPA via Python API

To programmatically force LingBot-Map to use the SDPA backend, pass use_sdpa=True when constructing your model instance. This approach works with any model class that accepts the flag, such as GCTStream:

import torch
from lingbot_map.models.gct_stream import GCTStream

# Build model with SDPA backend forced

model = GCTStream(
    img_size=518,
    patch_size=14,
    embed_dim=768,
    patch_embed="dinov2_vitl14_reg",
    pretrained_path="path/to/weights.pt",
    enable_3d_rope=True,
    kv_cache_sliding_window=64,
    kv_cache_scale_frames=8,
    use_sdpa=True,                 # Forces SDPA backend

)

# Inference runs with native PyTorch attention

dummy_imgs = torch.randn(1, 4, 3, 518, 378).cuda()
output = model(dummy_imgs)

When use_sdpa=True, the aggregator's GlobalBlockCls selector (lines 36-38 in stream.py) instantiates SDPABlock rather than FlashInferBlock, and the _init_kv_cache method initializes a dict-based cache instead of the FlashInfer paged cache.

Method 2: Force SDPA via Command-Line Interface

For CLI-based workflows using demo.py, add the --use_sdpa flag to force the SDPA backend without modifying source code. The argument parser (lines 82-84 in demo.py) exposes this flag:

parser.add_argument("--use_sdpa", action="store_true", default=False,
                    help="Use SDPA backend (no flashinfer needed). Default: FlashInfer")

Execute your inference script with the flag:

python demo.py \
    --model_path weights.pt \
    --image_folder ./my_dataset \
    --use_sdpa               # Forces SDPA backend

This sets use_sdpa=True internally when the model builder constructs the AggregatorStream instance.

Verifying the Active Backend

Confirm which backend is running by examining the log output from lingbot_map/aggregator/stream.py. The _init_kv_cache method emits distinct messages for each backend:

• FlashInfer active: INFO:lingbot_map.aggregator.stream:FlashInfer KV cache will be lazily initialized on first forward
• SDPA active: INFO:lingbot_map.aggregator.stream:SDPA KV cache initialized with X blocks

You can also verify programmatically by inspecting the block class type:

from lingbot_map.models.gct_stream import GCTStream

model = GCTStream(..., use_sdpa=True)

# Check the global block class inside the aggregator

block_type = type(model.aggregator.global_blocks[0]).__name__
print(block_type)   # prints "SDPABlock" when SDPA is forced

Technical Implementation Details

The backend switch affects three critical components:

1. Cache Manager: In lingbot_map/aggregator/stream.py, _init_kv_cache creates a dict object for SDPA mode versus a FlashInferKVCacheManager for standard mode.

2. Attention Block Selection: The GlobalBlockCls variable selects between SDPABlock and FlashInferBlock (defined in lingbot_map/layers/attention.py) based on the use_sdpa flag.

3. Model Wrapper: The GCTStream class in lingbot_map/models/gct_stream.py explicitly passes use_flashinfer=not self.use_sdpa and use_sdpa=self.use_sdpa to ensure consistency across the architecture.

Summary

• Force via Python: Pass use_sdpa=True to GCTStream or any compatible model constructor.
• Force via CLI: Append --use_sdpa when running demo.py.
• Default behavior: FlashInfer is used when use_sdpa is omitted or set to False.
• Verification: Check logs for "SDPA KV cache initialized" or inspect model.aggregator.global_blocks[0] for SDPABlock type.
• Key files: Configuration flows through lingbot_map/models/gct_stream.py into lingbot_map/aggregator/stream.py, with CLI support in demo.py.

Frequently Asked Questions

What is the difference between FlashInfer and SDPA backends?

FlashInfer utilizes the specialized FlashInfer library with optimized CUDA kernels and paged KV-cache management for maximum throughput. SDPA uses PyTorch's native torch.nn.functional.scaled_dot_product_attention with a simpler dictionary-based cache, requiring no additional dependencies beyond standard PyTorch.

When should I force SDPA instead of using FlashInfer?

Force SDPA when FlashInfer installation fails, when running on hardware without CUDA support, or when debugging attention mechanisms where FlashInfer's optimized kernels obscure intermediate values. SDPA provides broader compatibility at the cost of reduced memory efficiency.

How do I verify which backend is currently active?

Check the log output for initialization messages: SDPA mode logs "SDPA KV cache initialized with X blocks" while FlashInfer logs "FlashInfer KV cache will be lazily initialized". Alternatively, inspect type(model.aggregator.global_blocks[0]) which returns SDPABlock for SDPA mode and FlashInferBlock for FlashInfer mode.

Does using SDPA affect model accuracy or output quality?

No, both backends implement mathematically identical scaled-dot-product attention computations. The SDPA backend produces identical results to FlashInfer, differing only in implementation details and memory management strategies.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}