How Sliding Window Attention Works in LingBot-Map's AggregatorStream Module

Sliding window attention in LingBot-Map's AggregatorStream limits each video frame to attend only to a fixed-size temporal history, keeping GPU memory usage constant at O(window size) rather than growing linearly with video length through KV-cache eviction and causal masking.

The AggregatorStream module in the Robbyant/lingbot-map repository processes video streams frame-by-frame using a causal transformer architecture. To handle arbitrarily long videos without exhausting GPU memory, the implementation employs a sliding window attention mechanism that restricts each frame's attention to a configurable number of previous frames. This approach balances temporal context with memory efficiency by actively evicting stale KV-cache entries rather than retaining the full sequence history.

Configuration Parameters

The sliding window behavior is controlled during AggregatorStream initialization through several key parameters defined in lingbot_map/aggregator/stream.py (lines 34-46):

• sliding_window_size: Determines how many previous frames (or blocks) each frame can attend to
• kv_cache_sliding_window: Sets the KV-cache eviction threshold in tokens
• kv_cache_scale_frames: Specifies how many initial "scale frames" to preserve for global context
• kv_cache_cross_frame_special: Flags whether to retain special tokens (camera/register) across the window boundary

When you instantiate the stream processor, these values are stored as instance attributes and propagated through the block hierarchy:

from lingbot_map.aggregator.stream import AggregatorStream

agg = AggregatorStream(
    sliding_window_size=8,           # Attend to last 8 frames

    kv_cache_sliding_window=64,      # Evict KV entries beyond 64 tokens

    kv_cache_scale_frames=2,         # Always keep first 2 scale frames

    kv_cache_cross_frame_special=True # Preserve special tokens

)

Implementation Details

The sliding window mechanism operates through two distinct paths depending on whether the model runs in batch mode or streaming mode. Both implementations reside primarily in lingbot_map/layers/attention.py with coordination from lingbot_map/layers/block.py.

Batch Mode with Explicit Masking

When processing without a KV cache (kv_cache is None), the attention layer constructs an explicit binary mask in CausalAttention.forward (lines 72-92). The logic:

1. Calculates the number of frames: num_frames = N // frame_seqlen
2. For each frame index i, allows attention to the current frame plus the previous sliding_window_size * num_frame_per_block frames
3. Masks out all earlier positions with negative infinity before applying F.scaled_dot_product_attention

This masking approach ensures causal, temporally-local attention without requiring memory management of past states, though it recomputes attention for all historical frames each forward pass.

Streaming Mode with KV-Cache Eviction

In production streaming scenarios, the model avoids explicit masking and instead implements sliding window attention through KV-cache eviction in CausalAttention._apply_kv_cache_eviction_causal (lines 97-115):

1. Before computing attention, the method identifies entries older than sliding_window_frames = self.kv_cache_sliding_window
2. It optionally retains scale frames and special tokens based on kv_cache_cross_frame_special and kv_cache_include_scale_frames flags
3. The pruned KV cache contains only the valid attention window, effectively restricting the receptive field without masking overhead

The AggregatorStream passes the window parameters down through _build_blocks (lines 418-426 in lingbot_map/layers/block.py), ensuring each FlashInferBlock or SDPABlock receives the current sliding_window_size during its forward pass.

FlashInfer Optimization Path

When FlashInfer acceleration is enabled, the eviction logic moves to FlashInferAttention.forward (lines 374-394). Here, the FlashInferKVCacheManager calls evict_frames() with the same sliding_window and scale_frames parameters used in the SDPA path, maintaining identical behavior while leveraging paged cache optimizations for higher throughput.

Practical Implementation Example

The following example demonstrates sliding window attention in a simulated streaming scenario:

import torch
from lingbot_map.aggregator.stream import AggregatorStream

# Dummy video: 10 frames, each with 16×16 patches + 2 special tokens

frame_length = 16 * 16 + 2
batch = torch.randn(1, 10 * frame_length, 768)

# Initialize with 3-frame sliding window

aggregator = AggregatorStream(
    sliding_window_size=3,
    kv_cache_sliding_window=64,
    kv_cache_scale_frames=2,
)

# Process frame-by-frame (streaming simulation)

outputs = []
for i in range(10):
    frame = batch[:, i * frame_length:(i + 1) * frame_length, :]
    out = aggregator(frame)  # Internal KV eviction maintains window

    outputs.append(out)

result = torch.cat(outputs, dim=1)  # Shape: (1, 10, 768)

In this loop, the aggregator's memory consumption remains constant because it never stores more than sliding_window_size frames (plus the 2 scale frames) in the KV cache, regardless of how many frames have been processed.

Summary

• Sliding window attention restricts each frame's temporal context to a fixed-size history, preventing O(N) memory growth in long videos.
• Batch mode implements the window through explicit causal masking in CausalAttention.forward (lines 72-92).
• Streaming mode achieves the same effect by evicting stale KV-cache entries in _apply_kv_cache_eviction_causal (lines 97-115).
• FlashInfer support provides optimized paged-cache eviction in FlashInferAttention.forward (lines 374-394).
• The mechanism preserves scale frames and special tokens across the window boundary when configured, maintaining global context while limiting local attention.

Frequently Asked Questions

What is the difference between sliding_window_size and kv_cache_sliding_window?

sliding_window_size typically refers to the number of frames or blocks in the attention window, while kv_cache_sliding_window specifies the eviction threshold in tokens. The latter ensures the KV cache does not exceed a specific token count, which directly implements the temporal window constraint in streaming mode. Both values should be coordinated to maintain consistent attention spans across batch and streaming inference paths.

How does LingBot-Map handle the first few frames before the window fills?

According to the source code in lingbot_map/layers/attention.py, the attention mechanism remains causal even when the history is shorter than the window size. In batch mode, the mask construction logic naturally handles partial windows by only masking future positions. In streaming mode, the KV cache simply contains fewer entries until the buffer fills, allowing the model to attend to all available previous frames without special-case handling.

Can scale frames be attended to even if they fall outside the sliding window?

Yes. The kv_cache_scale_frames and kv_cache_cross_frame_special parameters in AggregatorStream configure the eviction logic to preserve designated scale frames and special tokens (camera/register) regardless of their age. As implemented in _apply_kv_cache_eviction_causal, these frames are explicitly excluded from eviction, ensuring the model maintains global scale information from video start while still limiting regular frame attention to the sliding window.

Why use KV-cache eviction instead of masking for streaming video?

KV-cache eviction provides superior memory efficiency and computational performance for long sequences. Building an explicit attention mask for every frame would require O(N²) memory and computation relative to sequence length, while physically removing old KV entries from the cache keeps memory constant at O(window size) and allows the attention computation to focus only on relevant tokens. This design choice enables LingBot-Map to process arbitrarily long video streams without degradation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}