# How the Geometric Context Transformer KV Cache Handles Sequences Longer Than 320 Frames

> Learn how the Geometric Context Transformer KV cache processes sequences longer than 320 frames using a spatial-temporal attention mechanism and sliding window to optimize memory.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: deep-dive
- Published: 2026-07-04

---

**The Geometric Context Transformer (GCT) employs a sliding-window eviction strategy combined with key-frame sparsity to process arbitrarily long video sequences while capping memory usage at roughly 72 frames regardless of total input length.**

The `Robbyant/lingbot-map` repository implements a specialized key-value (KV) cache mechanism designed specifically for streaming video inference. When processing sequences exceeding 320 frames, the GCT KV cache utilizes configurable sliding windows, selective key-frame retention, and special-token preservation to maintain bounded GPU memory consumption without sacrificing temporal coherence.

## KV Cache Architecture and Sliding Window Parameters

The GCT model initializes its cache structure in `GCTStream.__init__` within [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py) (lines 22-28). Each transformer block maintains a dictionary `kv_cache` storing tensors shaped `[B, H, T, D]` (batch, heads, token-time, head-dim), enabling causal attention to past frames without recomputing embeddings.

### Configurable Capacity Limits

The system enforces memory bounds through three critical parameters:

- **`kv_cache_sliding_window`** (default 64): Maximum number of recent non-scale frames retained in GPU memory
- **`kv_cache_scale_frames`** (default 8): Fixed scale frames providing bidirectional attention for depth estimation
- **`kv_cache_include_scale_frames`**: Boolean flag determining whether scale frames coexist with the sliding window or operate separately

These parameters are passed to the `AggregatorStream` and attention layers during model construction, establishing the hard upper bound of 72 total cached frames (64 + 8) regardless of video duration.

## Sliding-Window Eviction for Long Sequences

When the number of processed frames exceeds the configured capacity, the GCT triggers `_apply_kv_cache_eviction_causal` in [`lingbot_map/layers/attention.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/attention.py) (lines 297-350) immediately before attention computation.

### Pre-Attention Cache Trimming

The eviction logic executes the following steps:

```python
if num_cached_frames > sliding_window_frames + scale_frames:
    evict_start = scale_frames
    evict_end = num_cached_frames - sliding_window_frames
    # Remove frames between evict_start and evict_end

    # Retain scale frames (oldest) and most recent sliding window frames

```

This procedure removes the oldest non-scale frames from the middle of the sequence, then reconstructs the cache by concatenating the retained scale frames with the newest `sliding_window_frames` entries. Because eviction occurs before attention matrix construction, the causal mask requires no recomputation.

### Special Token Preservation

Even when evicting frame embeddings, the model preserves global context through `kv_cache_cross_frame_special`. When enabled, special tokens (camera tokens, register tokens, and scale tokens) from evicted frames transfer to auxiliary buffers named `k_{i}_special` and `v_{i}_special` (lines 18-28 in [`attention.py`](https://github.com/Robbyant/lingbot-map/blob/main/attention.py)). This ensures camera pose information and other cross-frame metadata remain accessible for downstream heads despite the sliding window constraint.

## Key-Frame Sparsity and Memory Optimization

To further reduce memory footprint during long sequences, the GCT implements selective KV storage via the `_set_skip_append(True)` method (lines 14-22 in [`gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream_window_v2.py)).

### Skip-Append Mechanism

During `inference_streaming`, the model evaluates each frame against `keyframe_interval` (default 4) and flow-based thresholds:

- **Keyframes**: Full KV tensors are computed, stored, and subject to sliding-window eviction
- **Non-keyframes**: The model sets `_skip_append` before storage, using the KV for immediate attention but discarding it afterward

With a keyframe interval of 4, effective cache size drops to approximately 18 frames (72 / 4), enabling processing of hour-long videos with minimal GPU memory.

### Rollback and Defer-Eviction

For frames initially classified as non-keyframes but later promoted based on optical flow thresholds, the model supports cache rollback and deferred eviction (lines 42-48 in [`gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream_window_v2.py)). This mechanism removes the previous append, executes the deferred eviction routine, and repopulates the cache with corrected keyframe data, ensuring temporal consistency without memory duplication.

## Practical Implementation

Creating a GCT model with optimized long-sequence parameters:

```python
from lingbot_map.models.gct_stream_window_v2 import GCTStream

model = GCTStream(
    img_size=518,
    patch_size=14,
    embed_dim=1024,
    enable_stream_inference=True,
    kv_cache_sliding_window=64,      # Keep at most 64 recent frames

    kv_cache_scale_frames=8,         # Always retain 8 scale frames

    kv_cache_include_scale_frames=True,
    keyframe_interval=4,             # Only every 4th frame stores KV

)

```

Processing video streams exceeding 320 frames:

```python

# video tensor shape: [N, 3, H, W] where N > 320

model.clean_kv_cache()  # Reset buffers (lines 88-97)

predictions = model.inference_streaming(
    images=video,
    keyframe_interval=4,
    output_device='cpu'
)

```

Monitoring cache statistics:

```python
info = model.get_kv_cache_info()
print(info)  # Output: {'num_cached_blocks': 12, 'cache_memory_mb': 14.5}

```

The `get_kv_cache_info()` method (lines 84-92) reports current tensor sizes and approximate memory footprints, useful for debugging memory usage during deployment.

## Summary

- **Hard memory cap**: The GCT KV cache enforces a strict limit of `kv_cache_sliding_window + kv_cache_scale_frames` frames (default 72), preventing out-of-memory errors on arbitrarily long videos
- **Eviction timing**: `_apply_kv_cache_eviction_causal` executes before attention in [`lingbot_map/layers/attention.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/attention.py), removing oldest non-scale frames while preserving geometric consistency
- **Key-frame sparsity**: The `_set_skip_append` mechanism reduces effective cache entries by 75% when using default intervals, processing thousands of frames with constant memory
- **Cross-frame persistence**: Special tokens survive eviction via dedicated `*_special` buffers, maintaining global camera pose information across the entire sequence
- **Rollback capability**: Deferred eviction and rollback support correct keyframe promotion without cache corruption

## Frequently Asked Questions

### What happens when the GCT KV cache reaches 320 frames?

The cache never accumulates 320 frames. At 72 frames (default), the sliding-window eviction automatically triggers, discarding the oldest non-scale frames while retaining the most recent 64 frames plus 8 fixed scale frames. This circular buffer behavior ensures constant memory usage regardless of video length.

### How does the sliding window affect temporal attention quality?

The model maintains causal attention to the most recent 64 frames plus 8 scale frames, providing approximately 2-3 seconds of temporal context at standard frame rates. This window captures sufficient motion history for depth estimation and camera pose prediction while preventing the quadratic memory growth typical of full-sequence attention.

### Can I adjust the KV cache size for longer temporal windows?

Yes, increase `kv_cache_sliding_window` when initializing `GCTStream`, though GPU memory consumption scales linearly with the window size. For extremely long dependencies, combine larger windows with decreased `keyframe_interval` to balance memory and temporal reach.

### What are scale frames and why are they exempt from eviction?

Scale frames provide bidirectional attention necessary for metric depth estimation and geometric consistency. They remain fixed at the beginning of the cache (indices 0-7) to anchor the global coordinate system, ensuring that even as the sliding window advances, the model maintains absolute scale references for accurate 3D reconstruction.