# Sliding Window Attention Mechanism in AggregatorStream: Implementation and Configuration Guide

> Understand the sliding window attention mechanism in AggregatorStream. Implement and configure it to limit temporal receptive fields and prevent KV-cache memory issues during streaming. Explore Robbyant/lingbot-map.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: how-to-guide
- Published: 2026-07-04

---

**The sliding window attention mechanism in `AggregatorStream` limits each video frame's temporal receptive field to a bounded set of past frames using the `sliding_window_size` parameter, preventing KV-cache memory explosion during indefinite streaming while maintaining causal dependencies.**

The `AggregatorStream` class in the Robbyant/lingbot-map repository implements this specialized attention pattern for streaming video-level vision transformers. By capping the number of historical frames that any given token can attend to, the system achieves constant memory complexity regardless of video duration.

## Configuring the Sliding Window in AggregatorStream

The sliding window behavior is controlled at the stream initialization level through the `sliding_window_size` argument. In [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py) (lines 34-44), the constructor accepts this parameter to limit temporal context.

When `sliding_window_size` is set to a positive integer, the mechanism restricts attention to that many blocks of past frames. A value of `-1` (the default) disables the window, allowing full causal attention across the entire sequence:

```python
from lingbot_map.aggregator.stream import AggregatorStream

# Initialize with a 6-block sliding window

stream = AggregatorStream(
    sliding_window_size=6,          # 6 blocks → limited temporal context

    kv_cache_sliding_window=64,     # Tokens per block in KV cache

    kv_cache_scale_frames=8,        # Preserve first 8 scale frames

    enable_3d_rope=True,
)

```

The configuration is stored in `self.sliding_window_size` (lines 71-73 of [`stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/stream.py)) and propagated downstream to the attention layers.

## Mask Generation in CausalAttention

The actual binary **sliding-window mask** is constructed within the `CausalAttention` class in [`lingbot_map/layers/attention.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/attention.py) (lines 172-190). During the forward pass, the method calculates which key-value pairs are reachable for each query token based on the sliding window configuration.

The mask computation interprets the token sequence as frames using the formula:

```

num_frames = N // frame_seqlen

```

For each frame `i`, the allowed key range spans from `window_start_frame` to the current frame, where:

```

window_start_frame = max(0, i - sliding_window_size * num_frame_per_block + 1)

```

This generates a boolean tensor named `sliding_mask` that is combined with the existing causal mask via a bitwise AND operation (`mask = mask & sliding_mask`). The resulting tensor instructs `torch.scaled_dot_product_attention` which historical tokens are visible to each query.

## KV Cache Eviction Strategy

To maintain memory bounds, the system evicts stale entries **before** computing attention. The method `_apply_kv_cache_eviction_causal` in [`lingbot_map/layers/attention.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/attention.py) (lines 997-1010) manages this pruning.

The eviction logic checks `num_cached_frames` and removes old frames when the cache exceeds `sliding_window_frames + scale_frames`:

```python

# Pseudocode representing the eviction logic

if num_cached_frames > sliding_window_frames + scale_frames:
    evict_start = scale_frames
    evict_end = num_cached_frames - sliding_window_frames
    # Remove frames [evict_start, evict_end) from KV cache

```

Special tokens (camera, register, scale) from evicted frames can be preserved using the `kv_cache_cross_frame_special` option. This eviction guarantees that the attention kernel only processes KV pairs within the sliding window, ensuring both computational efficiency and causal correctness.

## Architecture Propagation

The sliding window configuration flows through the block architecture via the `Block` class in [`lingbot_map/layers/block.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/block.py). The `forward` method (lines 77-82) passes `sliding_window_size` to the internal attention module, ensuring that `CausalAttention` and `FlashInferBlock` instances receive the window constraints.

For FlashInfer-backed execution, the physical memory management occurs in [`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py) (lines 70-90), where the paged KV cache drops pages outside the window boundary. This low-level eviction works in tandem with the high-level mask generation to enforce the window constraint consistently across different attention backends (SDPA and FlashInfer).

## Practical Implementation Examples

### Streaming Video Processing

The following example demonstrates instantiating an `AggregatorStream` with sliding window constraints for long-form video:

```python
from lingbot_map.aggregator.stream import AggregatorStream

# Create stream with 4-block attention window

stream = AggregatorStream(
    sliding_window_size=4,
    kv_cache_sliding_window=64,
    use_sdpa=False,  # Use FlashInfer for optimized KV cache

)

# Process tokens: [B, S_local, P, C] → [B, S_local, P, C]

output = stream(frame_tokens)

```

### Manual Sliding-Window Mask Construction

To understand the masking logic used internally, you can replicate the mask generation algorithm:

```python
import torch

def build_sliding_mask(num_frames, frame_seqlen, num_frame_per_block, sliding_window_size):
    """
    Reproduces the mask logic from CausalAttention (lines 172-190).
    Returns boolean mask of shape [1, 1, N, N].
    """
    N = num_frames * frame_seqlen
    mask = torch.zeros(1, 1, N, N, dtype=torch.bool)
    
    for i in range(num_frames):
        q_start = i * frame_seqlen
        q_end = (i + 1) * frame_seqlen
        
        win = sliding_window_size * num_frame_per_block
        w_start = max(0, i - win + 1) * frame_seqlen
        w_end = (i + 1) * frame_seqlen  # Causal: up to current frame

        
        mask[..., q_start:q_end, w_start:w_end] = True
    
    return mask

# Example: 10 frames, 1560 tokens each, window of 4 blocks

mask = build_sliding_mask(10, 1560, 1, 4)

```

### Cache Reset Between Sequences

When processing a new video, clear the KV cache to prevent cross-contamination:

```python

# Reset between sequences

stream.clean_kv_cache()

```

## Summary

- The **sliding window attention mechanism** enforces a bounded temporal receptive field via the `sliding_window_size` parameter in `AggregatorStream`, preventing unbounded memory growth during streaming.
- **Mask generation** occurs in `CausalAttention.forward` (lines 172-190), creating a binary mask that clips attention to recent frames based on `window_start_frame` calculations.
- **KV-cache eviction** is performed by `_apply_kv_cache_eviction_causal` (lines 997-1010) before attention computation, physically removing old entries while optionally preserving special tokens.
- The architecture propagates window constraints through `Block` wrappers down to FlashInfer's paged cache manager, ensuring consistent behavior across both SDPA and FlashInfer backends.

## Frequently Asked Questions

### How does sliding_window_size differ from kv_cache_sliding_window?

The `sliding_window_size` parameter controls the **attention mask** and determines how many past blocks each frame can attend to, affecting the model's temporal context window. In contrast, `kv_cache_sliding_window` specifies the physical **memory allocation** for the KV cache in tokens per block, governing how much GPU memory is reserved. While these values are typically coordinated, they operate at different layers: one controls computational visibility, the other manages memory footprint.

### Why is KV cache eviction performed before attention rather than after?

Evicting old KV entries **before** attention computation (as implemented in lines 997-1010 of [`attention.py`](https://github.com/Robbyant/lingbot-map/blob/main/attention.py)) ensures that the attention kernel never processes out-of-window keys, guaranteeing both memory efficiency and causal correctness. If eviction occurred after attention, the system would still compute unnecessary dot products with distant frames, defeating the computational savings purpose of the sliding window mechanism.

### Can special tokens persist outside the sliding window?

Yes. The eviction logic in `CausalAttention` supports the `kv_cache_cross_frame_special` flag, which preserves camera embeddings, register tokens, and scale frames even when their containing blocks fall outside the sliding window. This allows critical meta-information to persist indefinitely while still evicting redundant frame content, as handled in the eviction range calculation starting at line 997.

### What happens when sliding_window_size is set to -1?

Setting `sliding_window_size` to `-1` (the default) disables the window constraint, allowing full **causal attention** where each frame can attend to all previous frames in the sequence. While this maximizes temporal context, it causes KV-cache size to grow linearly with video length, making it unsuitable for indefinite streaming scenarios but appropriate for fixed-length video clips where memory is not constrained.