How the Sliding Window Limits Cache Size in LingBot-Map

LingBot-Map bounds GPU memory usage by maintaining a fixed-size sliding window that automatically evicts the oldest patch frames from the KV cache whenever new frames arrive, ensuring only the most recent N frames (default 64) remain resident in memory.

The Robbyant/lingbot-map repository implements a streaming transformer architecture where unbounded key-value (KV) caches would quickly exhaust GPU memory during long sequences. To prevent this, the system employs a sliding window mechanism that manages both logical attention masking and physical cache eviction. This design separates the conceptual window used for causal masking from the concrete window used for memory management, allowing precise control over cache size while maintaining model correctness.

Model Configuration Parameters

The sliding window behavior is governed by two distinct parameters passed to each transformer block during initialization.

sliding_window_size controls the logical window for causal masking, determining how many past frames a query can attend to. kv_cache_sliding_window defines the physical memory constraint, specifying exactly how many recent patch frames may remain alive in the KV cache. In lingbot_map/models/gct_stream.py, the default kv_cache_sliding_window is set to 64 at line 119, establishing the baseline memory bound for the recyclable patch stream.

These parameters propagate through the block wrapper in lingbot_map/layers/block.py (lines 271-279) and are exposed via the model API in lingbot_map/models/gct_stream_window.py (lines 146-164), allowing users to tune memory usage without modifying internal layer code.

FlashInfer KV-Cache Eviction Strategy

The FlashInferKVCacheManager splits the cache into a recyclable patch stream and a special append-only stream. When append_frame stores new KV pairs, the manager immediately calls evict_frames with the sliding_window argument to enforce the memory limit.

As implemented in lingbot_map/layers/flashinfer_cache.py (lines 10-11), the eviction logic trims live-window patch pages until their count is less than or equal to sliding_window. This guarantees that the total KV memory consumption never exceeds scale_frames + sliding_window + headroom, where headroom provides a small safety margin (lines 61-70). The oldest patch pages are recycled first, ensuring that only the most recent frames consume GPU memory.

Attention Masking Integration

Before the attention kernel executes, the layer constructs a sliding-window mask that restricts each query to attend only to keys within the current window. In lingbot_map/layers/attention.py (lines 172-188), when sliding_window_size > 0 and frame_seqlen is known, the code generates a boolean mask that is combined with the causal mask via mask = mask & sliding_mask.

Crucially, eviction occurs before attention computation. As documented in lines 242-304 of attention.py, the FlashInferKVCacheManager recycles outdated pages prior to mask application, ensuring that the attention layer never attempts to access evicted keys. This ordering maintains consistency between the physical cache state and the logical attention pattern.

End-to-End Processing Flow

When a transformer block processes a new frame, the system executes the following sequence:

1. Storage: FlashInferKVCacheManager.append_frame receives the new KV pairs and stores them in the recyclable patch stream.
2. Eviction: FlashInferKVCacheManager.evict_frames immediately removes any patch pages that exceed the kv_cache_sliding_window limit, recycling the oldest frames first.
3. Masking: The attention layer receives sliding_window_size (or kv_cache_sliding_window) and builds a mask that blocks attention to evicted positions.
4. Computation: The attention kernel processes only the retained frames, producing outputs that respect both the memory constraint and causal dependencies.

This pipeline ensures that memory usage remains constant regardless of sequence length, scaling only with the fixed window size rather than the total number of processed frames.

Configuration Example

You can instantiate a model with custom window constraints to reduce memory footprint for long sequences:

from lingbot_map.models.gct_stream import GCTStream

# Configure a model that keeps only the latest 32 patch frames in KV cache

model = GCTStream(
    num_blocks=12,
    sliding_window_size=-1,          # Use full causal masking (no window restriction)

    kv_cache_sliding_window=32,      # Limit physical cache to 32 recent frames

)

# During inference, eviction happens automatically:

# FlashInferKVCacheManager.evict_frames(block_idx, scale_frames=8, sliding_window=32)

# The attention layer masks out-of-window tokens automatically

# mask = mask & sliding_mask  (inside lingbot_map/layers/attention.py)

Summary

• LingBot-Map implements a fixed-size sliding window to bound KV cache memory usage in streaming scenarios.
• Two parameters control behavior: sliding_window_size for logical attention masking and kv_cache_sliding_window for physical memory limits (default 64).
• FlashInferKVCacheManager handles eviction in lingbot_map/layers/flashinfer_cache.py, recycling oldest patch pages before attention computation.
• Attention masking in lingbot_map/layers/attention.py ensures queries only attend to non-evicted frames by combining causal and sliding-window masks.
• Memory formula remains constant: scale_frames + sliding_window + headroom, independent of total sequence length.

Frequently Asked Questions

What is the difference between sliding_window_size and kv_cache_sliding_window?

sliding_window_size controls the logical attention pattern, determining how many past frames a query may attend to during the forward pass. kv_cache_sliding_window controls the physical memory allocation, dictating exactly how many recent patch frames remain stored in GPU memory. You can set these independently—for example, using a full causal mask (sliding_window_size=-1) while still limiting cache size to 32 frames to save memory.

Why does eviction happen before the attention computation?

Eviction occurs first to ensure consistency between the physical KV cache and the attention mask. If the attention layer attempted to access keys that had already been evicted from GPU memory, it would cause illegal memory access errors. By calling evict_frames before building the attention mask (as seen in lingbot_map/layers/attention.py lines 242-304), the system guarantees that the mask only references valid, resident cache entries.

How does the sliding window affect GPU memory usage?

The sliding window converts unbounded memory growth into constant space complexity. Instead of storing KV pairs for all previous frames, the system maintains a fixed buffer of size kv_cache_sliding_window (default 64) plus a small headroom. According to the implementation in lingbot_map/layers/flashinfer_cache.py (lines 61-70), total memory consumption scales as scale_frames + sliding_window + headroom, allowing the model to process infinite-length sequences with finite GPU memory.

What happens when I set sliding_window_size to -1?

Setting sliding_window_size to -1 disables the sliding-window mask, allowing full causal attention where each query attends to all previous keys. However, this does not disable cache eviction. To prevent out-of-memory errors, you must still configure kv_cache_sliding_window to a positive integer, or disable the KV cache manager entirely. The -1 value is handled in lingbot_map/layers/attention.py (lines 172-188) as a special case that skips sliding-mask generation while preserving other masking logic.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}