# How the Sliding Window Limits Cache Size in LingBot-Map

> Learn how LingBot-Map uses a sliding window to limit cache size in its KV cache. This technique evicts old frames, keeping memory usage bound and ensuring fast access to recent data.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: internals
- Published: 2026-07-03

---

**LingBot-Map bounds GPU memory usage by maintaining a fixed-size sliding window that automatically evicts the oldest patch frames from the KV cache whenever new frames arrive, ensuring only the most recent N frames (default 64) remain resident in memory.**

The Robbyant/lingbot-map repository implements a streaming transformer architecture where unbounded key-value (KV) caches would quickly exhaust GPU memory during long sequences. To prevent this, the system employs a **sliding window mechanism** that manages both logical attention masking and physical cache eviction. This design separates the conceptual window used for causal masking from the concrete window used for memory management, allowing precise control over cache size while maintaining model correctness.

## Model Configuration Parameters

The sliding window behavior is governed by two distinct parameters passed to each transformer block during initialization.

**`sliding_window_size`** controls the logical window for causal masking, determining how many past frames a query can attend to. **`kv_cache_sliding_window`** defines the physical memory constraint, specifying exactly how many recent patch frames may remain alive in the KV cache. In [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py), the default `kv_cache_sliding_window` is set to `64` at line 119, establishing the baseline memory bound for the recyclable patch stream.

These parameters propagate through the block wrapper in [`lingbot_map/layers/block.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/block.py) (lines 271-279) and are exposed via the model API in [`lingbot_map/models/gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py) (lines 146-164), allowing users to tune memory usage without modifying internal layer code.

## FlashInfer KV-Cache Eviction Strategy

The `FlashInferKVCacheManager` splits the cache into a **recyclable patch stream** and a **special append-only stream**. When `append_frame` stores new KV pairs, the manager immediately calls `evict_frames` with the `sliding_window` argument to enforce the memory limit.

As implemented in [`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py) (lines 10-11), the eviction logic trims live-window patch pages until their count is less than or equal to `sliding_window`. This guarantees that the total KV memory consumption never exceeds `scale_frames + sliding_window + headroom`, where `headroom` provides a small safety margin (lines 61-70). The oldest patch pages are recycled first, ensuring that only the most recent frames consume GPU memory.

## Attention Masking Integration

Before the attention kernel executes, the layer constructs a **sliding-window mask** that restricts each query to attend only to keys within the current window. In [`lingbot_map/layers/attention.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/attention.py) (lines 172-188), when `sliding_window_size > 0` and `frame_seqlen` is known, the code generates a boolean mask that is combined with the causal mask via `mask = mask & sliding_mask`.

Crucially, eviction occurs **before** attention computation. As documented in lines 242-304 of [`attention.py`](https://github.com/Robbyant/lingbot-map/blob/main/attention.py), the `FlashInferKVCacheManager` recycles outdated pages prior to mask application, ensuring that the attention layer never attempts to access evicted keys. This ordering maintains consistency between the physical cache state and the logical attention pattern.

## End-to-End Processing Flow

When a transformer block processes a new frame, the system executes the following sequence:

1. **Storage**: `FlashInferKVCacheManager.append_frame` receives the new KV pairs and stores them in the recyclable patch stream.
2. **Eviction**: `FlashInferKVCacheManager.evict_frames` immediately removes any patch pages that exceed the `kv_cache_sliding_window` limit, recycling the oldest frames first.
3. **Masking**: The attention layer receives `sliding_window_size` (or `kv_cache_sliding_window`) and builds a mask that blocks attention to evicted positions.
4. **Computation**: The attention kernel processes only the retained frames, producing outputs that respect both the memory constraint and causal dependencies.

This pipeline ensures that memory usage remains constant regardless of sequence length, scaling only with the fixed window size rather than the total number of processed frames.

## Configuration Example

You can instantiate a model with custom window constraints to reduce memory footprint for long sequences:

```python
from lingbot_map.models.gct_stream import GCTStream

# Configure a model that keeps only the latest 32 patch frames in KV cache

model = GCTStream(
    num_blocks=12,
    sliding_window_size=-1,          # Use full causal masking (no window restriction)

    kv_cache_sliding_window=32,      # Limit physical cache to 32 recent frames

)

# During inference, eviction happens automatically:

# FlashInferKVCacheManager.evict_frames(block_idx, scale_frames=8, sliding_window=32)

# The attention layer masks out-of-window tokens automatically

# mask = mask & sliding_mask  (inside lingbot_map/layers/attention.py)

```

## Summary

- **LingBot-Map** implements a fixed-size sliding window to bound KV cache memory usage in streaming scenarios.
- **Two parameters** control behavior: `sliding_window_size` for logical attention masking and `kv_cache_sliding_window` for physical memory limits (default 64).
- **FlashInferKVCacheManager** handles eviction in [`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py), recycling oldest patch pages before attention computation.
- **Attention masking** in [`lingbot_map/layers/attention.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/attention.py) ensures queries only attend to non-evicted frames by combining causal and sliding-window masks.
- **Memory formula** remains constant: `scale_frames + sliding_window + headroom`, independent of total sequence length.

## Frequently Asked Questions

### What is the difference between `sliding_window_size` and `kv_cache_sliding_window`?

`sliding_window_size` controls the logical attention pattern, determining how many past frames a query may attend to during the forward pass. `kv_cache_sliding_window` controls the physical memory allocation, dictating exactly how many recent patch frames remain stored in GPU memory. You can set these independently—for example, using a full causal mask (`sliding_window_size=-1`) while still limiting cache size to 32 frames to save memory.

### Why does eviction happen before the attention computation?

Eviction occurs first to ensure consistency between the physical KV cache and the attention mask. If the attention layer attempted to access keys that had already been evicted from GPU memory, it would cause illegal memory access errors. By calling `evict_frames` before building the attention mask (as seen in [`lingbot_map/layers/attention.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/attention.py) lines 242-304), the system guarantees that the mask only references valid, resident cache entries.

### How does the sliding window affect GPU memory usage?

The sliding window converts unbounded memory growth into constant space complexity. Instead of storing KV pairs for all previous frames, the system maintains a fixed buffer of size `kv_cache_sliding_window` (default 64) plus a small headroom. According to the implementation in [`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py) (lines 61-70), total memory consumption scales as `scale_frames + sliding_window + headroom`, allowing the model to process infinite-length sequences with finite GPU memory.

### What happens when I set `sliding_window_size` to -1?

Setting `sliding_window_size` to `-1` disables the sliding-window mask, allowing full causal attention where each query attends to all previous keys. However, this does not disable cache eviction. To prevent out-of-memory errors, you must still configure `kv_cache_sliding_window` to a positive integer, or disable the KV cache manager entirely. The `-1` value is handled in [`lingbot_map/layers/attention.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/attention.py) (lines 172-188) as a special case that skips sliding-mask generation while preserving other masking logic.