# How Sliding Window Attention Works in LingBot-Map's AggregatorStream Module

> Discover how sliding window attention in LingBot-Map's AggregatorStream optimizes GPU memory by limiting attention to a fixed temporal history. Learn more now.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: internals
- Published: 2026-06-29

---

**Sliding window attention in LingBot-Map's AggregatorStream limits each video frame to attend only to a fixed-size temporal history, keeping GPU memory usage constant at O(window size) rather than growing linearly with video length through KV-cache eviction and causal masking.**

The **AggregatorStream** module in the [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map) repository processes video streams frame-by-frame using a causal transformer architecture. To handle arbitrarily long videos without exhausting GPU memory, the implementation employs a **sliding window attention** mechanism that restricts each frame's attention to a configurable number of previous frames. This approach balances temporal context with memory efficiency by actively evicting stale KV-cache entries rather than retaining the full sequence history.

## Configuration Parameters

The sliding window behavior is controlled during `AggregatorStream` initialization through several key parameters defined in [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py) (lines 34-46):

- **`sliding_window_size`**: Determines how many previous frames (or blocks) each frame can attend to
- **`kv_cache_sliding_window`**: Sets the KV-cache eviction threshold in tokens
- **`kv_cache_scale_frames`**: Specifies how many initial "scale frames" to preserve for global context
- **`kv_cache_cross_frame_special`**: Flags whether to retain special tokens (camera/register) across the window boundary

When you instantiate the stream processor, these values are stored as instance attributes and propagated through the block hierarchy:

```python
from lingbot_map.aggregator.stream import AggregatorStream

agg = AggregatorStream(
    sliding_window_size=8,           # Attend to last 8 frames

    kv_cache_sliding_window=64,      # Evict KV entries beyond 64 tokens

    kv_cache_scale_frames=2,         # Always keep first 2 scale frames

    kv_cache_cross_frame_special=True # Preserve special tokens

)

```

## Implementation Details

The sliding window mechanism operates through two distinct paths depending on whether the model runs in batch mode or streaming mode. Both implementations reside primarily in [`lingbot_map/layers/attention.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/attention.py) with coordination from [`lingbot_map/layers/block.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/block.py).

### Batch Mode with Explicit Masking

When processing without a KV cache (`kv_cache is None`), the attention layer constructs an explicit binary mask in `CausalAttention.forward` (lines 72-92). The logic:

1. Calculates the number of frames: `num_frames = N // frame_seqlen`
2. For each frame index *i*, allows attention to the current frame plus the previous `sliding_window_size * num_frame_per_block` frames
3. Masks out all earlier positions with negative infinity before applying `F.scaled_dot_product_attention`

This masking approach ensures causal, temporally-local attention without requiring memory management of past states, though it recomputes attention for all historical frames each forward pass.

### Streaming Mode with KV-Cache Eviction

In production streaming scenarios, the model avoids explicit masking and instead implements sliding window attention through **KV-cache eviction** in `CausalAttention._apply_kv_cache_eviction_causal` (lines 97-115):

1. Before computing attention, the method identifies entries older than `sliding_window_frames = self.kv_cache_sliding_window`
2. It optionally retains scale frames and special tokens based on `kv_cache_cross_frame_special` and `kv_cache_include_scale_frames` flags
3. The pruned KV cache contains only the valid attention window, effectively restricting the receptive field without masking overhead

The `AggregatorStream` passes the window parameters down through `_build_blocks` (lines 418-426 in [`lingbot_map/layers/block.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/block.py)), ensuring each `FlashInferBlock` or `SDPABlock` receives the current `sliding_window_size` during its forward pass.

### FlashInfer Optimization Path

When FlashInfer acceleration is enabled, the eviction logic moves to `FlashInferAttention.forward` (lines 374-394). Here, the `FlashInferKVCacheManager` calls `evict_frames()` with the same `sliding_window` and `scale_frames` parameters used in the SDPA path, maintaining identical behavior while leveraging paged cache optimizations for higher throughput.

## Practical Implementation Example

The following example demonstrates sliding window attention in a simulated streaming scenario:

```python
import torch
from lingbot_map.aggregator.stream import AggregatorStream

# Dummy video: 10 frames, each with 16×16 patches + 2 special tokens

frame_length = 16 * 16 + 2
batch = torch.randn(1, 10 * frame_length, 768)

# Initialize with 3-frame sliding window

aggregator = AggregatorStream(
    sliding_window_size=3,
    kv_cache_sliding_window=64,
    kv_cache_scale_frames=2,
)

# Process frame-by-frame (streaming simulation)

outputs = []
for i in range(10):
    frame = batch[:, i * frame_length:(i + 1) * frame_length, :]
    out = aggregator(frame)  # Internal KV eviction maintains window

    outputs.append(out)

result = torch.cat(outputs, dim=1)  # Shape: (1, 10, 768)

```

In this loop, the aggregator's memory consumption remains constant because it never stores more than `sliding_window_size` frames (plus the 2 scale frames) in the KV cache, regardless of how many frames have been processed.

## Summary

- **Sliding window attention** restricts each frame's temporal context to a fixed-size history, preventing O(N) memory growth in long videos.
- **Batch mode** implements the window through explicit causal masking in `CausalAttention.forward` (lines 72-92).
- **Streaming mode** achieves the same effect by evicting stale KV-cache entries in `_apply_kv_cache_eviction_causal` (lines 97-115).
- **FlashInfer support** provides optimized paged-cache eviction in `FlashInferAttention.forward` (lines 374-394).
- The mechanism preserves **scale frames** and **special tokens** across the window boundary when configured, maintaining global context while limiting local attention.

## Frequently Asked Questions

### What is the difference between sliding_window_size and kv_cache_sliding_window?

**`sliding_window_size`** typically refers to the number of frames or blocks in the attention window, while **`kv_cache_sliding_window`** specifies the eviction threshold in tokens. The latter ensures the KV cache does not exceed a specific token count, which directly implements the temporal window constraint in streaming mode. Both values should be coordinated to maintain consistent attention spans across batch and streaming inference paths.

### How does LingBot-Map handle the first few frames before the window fills?

According to the source code in [`lingbot_map/layers/attention.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/attention.py), the attention mechanism remains causal even when the history is shorter than the window size. In batch mode, the mask construction logic naturally handles partial windows by only masking future positions. In streaming mode, the KV cache simply contains fewer entries until the buffer fills, allowing the model to attend to all available previous frames without special-case handling.

### Can scale frames be attended to even if they fall outside the sliding window?

Yes. The `kv_cache_scale_frames` and `kv_cache_cross_frame_special` parameters in `AggregatorStream` configure the eviction logic to preserve designated scale frames and special tokens (camera/register) regardless of their age. As implemented in `_apply_kv_cache_eviction_causal`, these frames are explicitly excluded from eviction, ensuring the model maintains global scale information from video start while still limiting regular frame attention to the sliding window.

### Why use KV-cache eviction instead of masking for streaming video?

KV-cache eviction provides superior memory efficiency and computational performance for long sequences. Building an explicit attention mask for every frame would require O(N²) memory and computation relative to sequence length, while physically removing old KV entries from the cache keeps memory constant at O(window size) and allows the attention computation to focus only on relevant tokens. This design choice enables LingBot-Map to process arbitrarily long video streams without degradation.