Sliding Window Attention Mechanism in AggregatorStream: Implementation and Configuration Guide

The sliding window attention mechanism in AggregatorStream limits each video frame's temporal receptive field to a bounded set of past frames using the sliding_window_size parameter, preventing KV-cache memory explosion during indefinite streaming while maintaining causal dependencies.

The AggregatorStream class in the Robbyant/lingbot-map repository implements this specialized attention pattern for streaming video-level vision transformers. By capping the number of historical frames that any given token can attend to, the system achieves constant memory complexity regardless of video duration.

Configuring the Sliding Window in AggregatorStream

The sliding window behavior is controlled at the stream initialization level through the sliding_window_size argument. In lingbot_map/aggregator/stream.py (lines 34-44), the constructor accepts this parameter to limit temporal context.

When sliding_window_size is set to a positive integer, the mechanism restricts attention to that many blocks of past frames. A value of -1 (the default) disables the window, allowing full causal attention across the entire sequence:

from lingbot_map.aggregator.stream import AggregatorStream

# Initialize with a 6-block sliding window

stream = AggregatorStream(
    sliding_window_size=6,          # 6 blocks → limited temporal context

    kv_cache_sliding_window=64,     # Tokens per block in KV cache

    kv_cache_scale_frames=8,        # Preserve first 8 scale frames

    enable_3d_rope=True,
)

The configuration is stored in self.sliding_window_size (lines 71-73 of stream.py) and propagated downstream to the attention layers.

Mask Generation in CausalAttention

The actual binary sliding-window mask is constructed within the CausalAttention class in lingbot_map/layers/attention.py (lines 172-190). During the forward pass, the method calculates which key-value pairs are reachable for each query token based on the sliding window configuration.

The mask computation interprets the token sequence as frames using the formula:

num_frames = N // frame_seqlen

For each frame i, the allowed key range spans from window_start_frame to the current frame, where:

window_start_frame = max(0, i - sliding_window_size * num_frame_per_block + 1)

This generates a boolean tensor named sliding_mask that is combined with the existing causal mask via a bitwise AND operation (mask = mask & sliding_mask). The resulting tensor instructs torch.scaled_dot_product_attention which historical tokens are visible to each query.

KV Cache Eviction Strategy

To maintain memory bounds, the system evicts stale entries before computing attention. The method _apply_kv_cache_eviction_causal in lingbot_map/layers/attention.py (lines 997-1010) manages this pruning.

The eviction logic checks num_cached_frames and removes old frames when the cache exceeds sliding_window_frames + scale_frames:

# Pseudocode representing the eviction logic

if num_cached_frames > sliding_window_frames + scale_frames:
    evict_start = scale_frames
    evict_end = num_cached_frames - sliding_window_frames
    # Remove frames [evict_start, evict_end) from KV cache

Special tokens (camera, register, scale) from evicted frames can be preserved using the kv_cache_cross_frame_special option. This eviction guarantees that the attention kernel only processes KV pairs within the sliding window, ensuring both computational efficiency and causal correctness.

Architecture Propagation

The sliding window configuration flows through the block architecture via the Block class in lingbot_map/layers/block.py. The forward method (lines 77-82) passes sliding_window_size to the internal attention module, ensuring that CausalAttention and FlashInferBlock instances receive the window constraints.

For FlashInfer-backed execution, the physical memory management occurs in lingbot_map/layers/flashinfer_cache.py (lines 70-90), where the paged KV cache drops pages outside the window boundary. This low-level eviction works in tandem with the high-level mask generation to enforce the window constraint consistently across different attention backends (SDPA and FlashInfer).

Practical Implementation Examples

Streaming Video Processing

The following example demonstrates instantiating an AggregatorStream with sliding window constraints for long-form video:

from lingbot_map.aggregator.stream import AggregatorStream

# Create stream with 4-block attention window

stream = AggregatorStream(
    sliding_window_size=4,
    kv_cache_sliding_window=64,
    use_sdpa=False,  # Use FlashInfer for optimized KV cache

)

# Process tokens: [B, S_local, P, C] → [B, S_local, P, C]

output = stream(frame_tokens)

Manual Sliding-Window Mask Construction

To understand the masking logic used internally, you can replicate the mask generation algorithm:

import torch

def build_sliding_mask(num_frames, frame_seqlen, num_frame_per_block, sliding_window_size):
    """
    Reproduces the mask logic from CausalAttention (lines 172-190).
    Returns boolean mask of shape [1, 1, N, N].
    """
    N = num_frames * frame_seqlen
    mask = torch.zeros(1, 1, N, N, dtype=torch.bool)
    
    for i in range(num_frames):
        q_start = i * frame_seqlen
        q_end = (i + 1) * frame_seqlen
        
        win = sliding_window_size * num_frame_per_block
        w_start = max(0, i - win + 1) * frame_seqlen
        w_end = (i + 1) * frame_seqlen  # Causal: up to current frame

        
        mask[..., q_start:q_end, w_start:w_end] = True
    
    return mask

# Example: 10 frames, 1560 tokens each, window of 4 blocks

mask = build_sliding_mask(10, 1560, 1, 4)

Cache Reset Between Sequences

When processing a new video, clear the KV cache to prevent cross-contamination:

# Reset between sequences

stream.clean_kv_cache()

Summary

• The sliding window attention mechanism enforces a bounded temporal receptive field via the sliding_window_size parameter in AggregatorStream, preventing unbounded memory growth during streaming.
• Mask generation occurs in CausalAttention.forward (lines 172-190), creating a binary mask that clips attention to recent frames based on window_start_frame calculations.
• KV-cache eviction is performed by _apply_kv_cache_eviction_causal (lines 997-1010) before attention computation, physically removing old entries while optionally preserving special tokens.
• The architecture propagates window constraints through Block wrappers down to FlashInfer's paged cache manager, ensuring consistent behavior across both SDPA and FlashInfer backends.

Frequently Asked Questions

How does sliding_window_size differ from kv_cache_sliding_window?

The sliding_window_size parameter controls the attention mask and determines how many past blocks each frame can attend to, affecting the model's temporal context window. In contrast, kv_cache_sliding_window specifies the physical memory allocation for the KV cache in tokens per block, governing how much GPU memory is reserved. While these values are typically coordinated, they operate at different layers: one controls computational visibility, the other manages memory footprint.

Why is KV cache eviction performed before attention rather than after?

Evicting old KV entries before attention computation (as implemented in lines 997-1010 of attention.py) ensures that the attention kernel never processes out-of-window keys, guaranteeing both memory efficiency and causal correctness. If eviction occurred after attention, the system would still compute unnecessary dot products with distant frames, defeating the computational savings purpose of the sliding window mechanism.

Can special tokens persist outside the sliding window?

Yes. The eviction logic in CausalAttention supports the kv_cache_cross_frame_special flag, which preserves camera embeddings, register tokens, and scale frames even when their containing blocks fall outside the sliding window. This allows critical meta-information to persist indefinitely while still evicting redundant frame content, as handled in the eviction range calculation starting at line 997.

What happens when sliding_window_size is set to -1?

Setting sliding_window_size to -1 (the default) disables the window constraint, allowing full causal attention where each frame can attend to all previous frames in the sequence. While this maximizes temporal context, it causes KV-cache size to grow linearly with video length, making it unsuitable for indefinite streaming scenarios but appropriate for fixed-length video clips where memory is not constrained.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}