# LingBot-Map Streaming Framework: Architecture for Real-Time Video Processing

> Explore the LingBot-Map streaming framework architecture for real-time video processing. Discover its causal sliding-window transformer and paged KV cache for efficient inference and incremental updates.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: architecture
- Published: 2026-07-06

---

**LingBot-Map processes video streams using a causal sliding-window transformer with a paged KV cache, enabling real-time inference by alternating between frame-wise and cross-frame attention while incrementally updating keys and values without recomputing past states.**

The **LingBot-Map streaming framework** provides a modular architecture for continuous video understanding in the `Robbyant/lingbot-map` repository. Built around a causal transformer design, the system maintains bounded memory usage through sliding-window attention and paginated caching, supporting both FlashInfer-accelerated and native PyTorch backends. This framework processes arbitrarily long video sequences by treating each frame incrementally while preserving temporal dependencies through specialized positional embeddings and dedicated cache management.

## Core Components of the LingBot-Map Streaming Framework

The framework consists of tightly coupled Python modules that handle orchestration, caching, and attention mechanisms. Each component serves a specific role in maintaining the causal, real-time processing pipeline.

### AggregatorStream and AggregatorBase

The `AggregatorStream` class in [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py) serves as the primary orchestration layer for streaming inference. It manages the sliding-window logic, decides between FlashInfer and native SDPA (Scaled Dot-Product Attention) backends, and coordinates the processing loop through `_process_causal_stream` and `_process_global_attention`.

The `AggregatorBase` class in [`lingbot_map/aggregator/base.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/base.py) provides shared functionality across all aggregator variants. It handles patch embedding via DINOv2, constructs transformer blocks, manages special token preparation through `_prepare_special_tokens`, and implements the generic forward scaffolding that alternates between frame and global attention blocks.

### FlashInferKVCacheManager and Transformer Blocks

At the heart of the inference engine lies the `FlashInferKVCacheManager` in [`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py). This component implements a two-stream paged KV cache (separate streams for patch and special tokens) that enables **incremental attention** without recomputing past keys and values. The manager supports both FA2 (non-power-of-2 pages) and FA3 (SM90) backends, and handles frame eviction through `evict_frames` while preserving scale and special tokens.

The transformer blocks are defined in [`lingbot_map/layers/block.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/block.py) and include three variants:

- **`Block`**: The base transformer block implementation
- **`FlashInferBlock`**: Utilizes the KV cache manager for accelerated attention
- **`SDPABlock`**: Falls back to native PyTorch SDPA when FlashInfer is unavailable

### Positional Encoding and Special Tokens

Spatial and temporal reasoning relies on rotary position embeddings defined in [`lingbot_map/layers/rope.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/rope.py). The `RotaryPositionEmbedding2D` class provides spatial encoding for patch tokens, while `WanRotaryPosEmbed` extends this to three dimensions for streaming scenarios. When `enable_3d_rope=True`, the framework generates temporal positions via `_get_3d_positions_streaming`, allowing the model to reason about absolute frame order.

Special tokens play a crucial role in the streaming pipeline. As implemented in `_setup_special_tokens` within [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py), the system prepends camera tokens, register tokens, and scale tokens to each frame. The scale token is particularly important for causal scale estimation and remains cached while patch tokens may be evicted.

## How the Streaming Pipeline Works

The LingBot-Map streaming framework processes video through five distinct stages that maintain causal dependencies while managing memory constraints.

### 1. Initialization

During `AggregatorStream.__init__`, the system parses streaming-specific hyperparameters including `sliding_window_size` (default 64), `num_frame_for_scale`, and KV-cache flags. The `_init_kv_cache` method instantiates either a `FlashInferKVCacheManager` or dictionary-based cache depending on the `use_sdpa` flag and hardware availability.

### 2. Embedding

The `AggregatorBase._embed_images` method normalizes input tensors and runs the DINOv2 patch embedder to convert frames into token sequences. The `_prepare_special_tokens` function concatenates camera, register, and scale tokens to the patch embeddings, creating the full input representation for each frame.

### 3. Alternating Attention

The forward loop in `AggregatorBase.forward` alternates between two block types:

- **Frame blocks**: Perform pure per-frame self-attention within individual frames
- **Global blocks**: Handle cross-frame attention to aggregate information across the temporal window

For streaming mode, `_process_global_attention` delegates to `AggregatorStream._process_causal_stream`, which implements the sliding-window logic.

### 4. KV Cache Updates

Inside `_process_causal_stream`, each global block either calls `FlashInferBlock` (which appends current frame K/V tensors via `FlashInferKVCacheManager.append_frame`) or falls back to the dict-based `SDPABlock` cache. After processing, the cache manager may evict old patch pages using `evict_frames` while preserving special and scale token pages permanently.

### 5. Output Generation

After each attention group, the model concatenates frame and global intermediate outputs using `torch.cat([frame, global], dim=-1)`. This produces a list of per-block feature maps ready for downstream tasks such as depth prediction or SLAM, with the patch-start index indicating where patch tokens begin in the sequence.

## Implementation Examples

### Basic CPU-Only Streaming

For environments without GPU acceleration, the framework automatically falls back to native SDPA:

```python
import torch
from lingbot_map.aggregator.stream import AggregatorStream

# Initialize with default settings - automatically selects backend

stream_agg = AggregatorStream(
    sliding_window_size=64,          # Retain last 64 frames in KV window

    num_frame_for_scale=8,           # Permanently cache 8 scale frames

    enable_3d_rope=False,           # Disable temporal RoPE

    use_sdpa=False,                  # Attempt FlashInfer first

)

# Dummy video: batch=1, 10 frames, RGB 224x224

videos = torch.rand(1, 10, 3, 224, 224)

# Forward pass returns block outputs and patch index

features, patch_start = stream_agg(videos)

print(f"Generated {len(features)} block outputs")
print(f"Feature shape: {features[0].shape}")  # [1, 10, patches, 2*embed_dim]

```

### Advanced Streaming with 3D RoPE

Enable temporal reasoning for long sequences requiring absolute frame positioning:

```python
stream_agg = AggregatorStream(
    sliding_window_size=32,
    num_frame_for_scale=4,
    enable_3d_rope=True,          # Enable temporal rotary embeddings

    max_frame_num=1024,           # Precompute RoPE for up to 1024 frames

    use_sdpa=False,
)

# Simulate live camera feed

for i in range(100):
    frame = torch.rand(1, 1, 3, 224, 224)
    feats, _ = stream_agg(frame)
    # Process hierarchical features for current frame

```

### Monitoring Cache State

Debug memory usage by inspecting the KV cache manager directly:

```python
mgr = stream_agg._get_flashinfer_manager(device='cuda', dtype=torch.bfloat16)
stats = mgr.get_cache_stats()
print(f"Active frames: {stats['frame_count']}")
print(f"Available pages: {stats['free_pages']}")

```

## Summary

- **LingBot-Map** implements a causal streaming transformer that processes video incrementally using a paged KV cache architecture.
- The **`AggregatorStream`** class orchestrates sliding-window attention and backend selection between FlashInfer and native SDPA.
- **`FlashInferKVCacheManager`** enables efficient incremental attention by caching keys and values across frames while supporting eviction policies.
- **Special tokens** (camera, register, scale) are prepended to frames and persist in cache while patch tokens follow the sliding-window constraints.
- **3D RoPE** (`WanRotaryPosEmbed`) extends spatial encoding to include temporal dimensions when `enable_3d_rope` is activated.
- The framework automatically manages memory through configurable `sliding_window_size` parameters and explicit cache eviction in [`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py).

## Frequently Asked Questions

### How does the sliding window mechanism manage long video sequences?

The sliding window constrains memory usage by evicting older frame patches while preserving special tokens. In `FlashInferKVCacheManager.evict_frames`, frames exceeding the `sliding_window_size` threshold are removed from the patch-page pool, while scale and special token pages remain append-only. This ensures the model maintains context through permanently cached scale tokens while bounding the computational cost of cross-frame attention to the most recent window.

### What is the difference between frame blocks and global blocks?

Frame blocks perform self-attention exclusively within individual frames, processing spatial relationships without temporal context. Global blocks handle cross-frame attention, allowing information to flow between frames in the current sliding window. In `AggregatorBase.forward`, the architecture alternates between these block types, with `AggregatorStream._process_causal_stream` managing how global blocks access the KV cache incrementally rather than recomputing full attention matrices.

### When should 3D RoPE be enabled in the streaming framework?

Enable 3D RoPE (`enable_3d_rope=True`) when the model requires explicit temporal positioning beyond relative frame ordering. According to `_get_3d_positions_streaming` in [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py), this feature generates absolute temporal rotary embeddings that help the model distinguish between frame positions in long sequences. This is particularly useful for tasks requiring precise temporal reasoning, though it increases memory usage for the position encoding tables.

### How does FlashInferKVCacheManager optimize inference performance?

The manager implements a paginated memory layout that supports non-power-of-2 page sizes (FA2) and SM90 architectures (FA3), allowing efficient append operations via `append_frame` without tensor concatenation overhead. By maintaining separate streams for patch and special tokens, it eliminates unnecessary memory copies during frame eviction. When FlashInfer is unavailable, the framework transparently falls back to `SDPABlock` with dictionary-based caching, ensuring functional consistency across hardware configurations.