LingBot-Map Streaming Framework: Architecture for Real-Time Video Processing

LingBot-Map processes video streams using a causal sliding-window transformer with a paged KV cache, enabling real-time inference by alternating between frame-wise and cross-frame attention while incrementally updating keys and values without recomputing past states.

The LingBot-Map streaming framework provides a modular architecture for continuous video understanding in the Robbyant/lingbot-map repository. Built around a causal transformer design, the system maintains bounded memory usage through sliding-window attention and paginated caching, supporting both FlashInfer-accelerated and native PyTorch backends. This framework processes arbitrarily long video sequences by treating each frame incrementally while preserving temporal dependencies through specialized positional embeddings and dedicated cache management.

Core Components of the LingBot-Map Streaming Framework

The framework consists of tightly coupled Python modules that handle orchestration, caching, and attention mechanisms. Each component serves a specific role in maintaining the causal, real-time processing pipeline.

AggregatorStream and AggregatorBase

The AggregatorStream class in lingbot_map/aggregator/stream.py serves as the primary orchestration layer for streaming inference. It manages the sliding-window logic, decides between FlashInfer and native SDPA (Scaled Dot-Product Attention) backends, and coordinates the processing loop through _process_causal_stream and _process_global_attention.

The AggregatorBase class in lingbot_map/aggregator/base.py provides shared functionality across all aggregator variants. It handles patch embedding via DINOv2, constructs transformer blocks, manages special token preparation through _prepare_special_tokens, and implements the generic forward scaffolding that alternates between frame and global attention blocks.

FlashInferKVCacheManager and Transformer Blocks

At the heart of the inference engine lies the FlashInferKVCacheManager in lingbot_map/layers/flashinfer_cache.py. This component implements a two-stream paged KV cache (separate streams for patch and special tokens) that enables incremental attention without recomputing past keys and values. The manager supports both FA2 (non-power-of-2 pages) and FA3 (SM90) backends, and handles frame eviction through evict_frames while preserving scale and special tokens.

The transformer blocks are defined in lingbot_map/layers/block.py and include three variants:

• Block: The base transformer block implementation
• FlashInferBlock: Utilizes the KV cache manager for accelerated attention
• SDPABlock: Falls back to native PyTorch SDPA when FlashInfer is unavailable

Positional Encoding and Special Tokens

Spatial and temporal reasoning relies on rotary position embeddings defined in lingbot_map/layers/rope.py. The RotaryPositionEmbedding2D class provides spatial encoding for patch tokens, while WanRotaryPosEmbed extends this to three dimensions for streaming scenarios. When enable_3d_rope=True, the framework generates temporal positions via _get_3d_positions_streaming, allowing the model to reason about absolute frame order.

Special tokens play a crucial role in the streaming pipeline. As implemented in _setup_special_tokens within lingbot_map/aggregator/stream.py, the system prepends camera tokens, register tokens, and scale tokens to each frame. The scale token is particularly important for causal scale estimation and remains cached while patch tokens may be evicted.

How the Streaming Pipeline Works

The LingBot-Map streaming framework processes video through five distinct stages that maintain causal dependencies while managing memory constraints.

1. Initialization

During AggregatorStream.__init__, the system parses streaming-specific hyperparameters including sliding_window_size (default 64), num_frame_for_scale, and KV-cache flags. The _init_kv_cache method instantiates either a FlashInferKVCacheManager or dictionary-based cache depending on the use_sdpa flag and hardware availability.

2. Embedding

The AggregatorBase._embed_images method normalizes input tensors and runs the DINOv2 patch embedder to convert frames into token sequences. The _prepare_special_tokens function concatenates camera, register, and scale tokens to the patch embeddings, creating the full input representation for each frame.

3. Alternating Attention

The forward loop in AggregatorBase.forward alternates between two block types:

• Frame blocks: Perform pure per-frame self-attention within individual frames
• Global blocks: Handle cross-frame attention to aggregate information across the temporal window

For streaming mode, _process_global_attention delegates to AggregatorStream._process_causal_stream, which implements the sliding-window logic.

4. KV Cache Updates

Inside _process_causal_stream, each global block either calls FlashInferBlock (which appends current frame K/V tensors via FlashInferKVCacheManager.append_frame) or falls back to the dict-based SDPABlock cache. After processing, the cache manager may evict old patch pages using evict_frames while preserving special and scale token pages permanently.

5. Output Generation

After each attention group, the model concatenates frame and global intermediate outputs using torch.cat([frame, global], dim=-1). This produces a list of per-block feature maps ready for downstream tasks such as depth prediction or SLAM, with the patch-start index indicating where patch tokens begin in the sequence.

Implementation Examples

Basic CPU-Only Streaming

For environments without GPU acceleration, the framework automatically falls back to native SDPA:

import torch
from lingbot_map.aggregator.stream import AggregatorStream

# Initialize with default settings - automatically selects backend

stream_agg = AggregatorStream(
    sliding_window_size=64,          # Retain last 64 frames in KV window

    num_frame_for_scale=8,           # Permanently cache 8 scale frames

    enable_3d_rope=False,           # Disable temporal RoPE

    use_sdpa=False,                  # Attempt FlashInfer first

)

# Dummy video: batch=1, 10 frames, RGB 224x224

videos = torch.rand(1, 10, 3, 224, 224)

# Forward pass returns block outputs and patch index

features, patch_start = stream_agg(videos)

print(f"Generated {len(features)} block outputs")
print(f"Feature shape: {features[0].shape}")  # [1, 10, patches, 2*embed_dim]

Advanced Streaming with 3D RoPE

Enable temporal reasoning for long sequences requiring absolute frame positioning:

stream_agg = AggregatorStream(
    sliding_window_size=32,
    num_frame_for_scale=4,
    enable_3d_rope=True,          # Enable temporal rotary embeddings

    max_frame_num=1024,           # Precompute RoPE for up to 1024 frames

    use_sdpa=False,
)

# Simulate live camera feed

for i in range(100):
    frame = torch.rand(1, 1, 3, 224, 224)
    feats, _ = stream_agg(frame)
    # Process hierarchical features for current frame

Monitoring Cache State

Debug memory usage by inspecting the KV cache manager directly:

mgr = stream_agg._get_flashinfer_manager(device='cuda', dtype=torch.bfloat16)
stats = mgr.get_cache_stats()
print(f"Active frames: {stats['frame_count']}")
print(f"Available pages: {stats['free_pages']}")

Summary

• LingBot-Map implements a causal streaming transformer that processes video incrementally using a paged KV cache architecture.
• The AggregatorStream class orchestrates sliding-window attention and backend selection between FlashInfer and native SDPA.
• FlashInferKVCacheManager enables efficient incremental attention by caching keys and values across frames while supporting eviction policies.
• Special tokens (camera, register, scale) are prepended to frames and persist in cache while patch tokens follow the sliding-window constraints.
• 3D RoPE (WanRotaryPosEmbed) extends spatial encoding to include temporal dimensions when enable_3d_rope is activated.
• The framework automatically manages memory through configurable sliding_window_size parameters and explicit cache eviction in lingbot_map/layers/flashinfer_cache.py.

Frequently Asked Questions

How does the sliding window mechanism manage long video sequences?

The sliding window constrains memory usage by evicting older frame patches while preserving special tokens. In FlashInferKVCacheManager.evict_frames, frames exceeding the sliding_window_size threshold are removed from the patch-page pool, while scale and special token pages remain append-only. This ensures the model maintains context through permanently cached scale tokens while bounding the computational cost of cross-frame attention to the most recent window.

What is the difference between frame blocks and global blocks?

Frame blocks perform self-attention exclusively within individual frames, processing spatial relationships without temporal context. Global blocks handle cross-frame attention, allowing information to flow between frames in the current sliding window. In AggregatorBase.forward, the architecture alternates between these block types, with AggregatorStream._process_causal_stream managing how global blocks access the KV cache incrementally rather than recomputing full attention matrices.

When should 3D RoPE be enabled in the streaming framework?

Enable 3D RoPE (enable_3d_rope=True) when the model requires explicit temporal positioning beyond relative frame ordering. According to _get_3d_positions_streaming in lingbot_map/aggregator/stream.py, this feature generates absolute temporal rotary embeddings that help the model distinguish between frame positions in long sequences. This is particularly useful for tasks requiring precise temporal reasoning, though it increases memory usage for the position encoding tables.

How does FlashInferKVCacheManager optimize inference performance?

The manager implements a paginated memory layout that supports non-power-of-2 page sizes (FA2) and SM90 architectures (FA3), allowing efficient append operations via append_frame without tensor concatenation overhead. By maintaining separate streams for patch and special tokens, it eliminates unnecessary memory copies during frame eviction. When FlashInfer is unavailable, the framework transparently falls back to SDPABlock with dictionary-based caching, ensuring functional consistency across hardware configurations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}