How the Geometric Context Transformer KV Cache Handles Sequences Longer Than 320 Frames

The Geometric Context Transformer (GCT) employs a sliding-window eviction strategy combined with key-frame sparsity to process arbitrarily long video sequences while capping memory usage at roughly 72 frames regardless of total input length.

The Robbyant/lingbot-map repository implements a specialized key-value (KV) cache mechanism designed specifically for streaming video inference. When processing sequences exceeding 320 frames, the GCT KV cache utilizes configurable sliding windows, selective key-frame retention, and special-token preservation to maintain bounded GPU memory consumption without sacrificing temporal coherence.

KV Cache Architecture and Sliding Window Parameters

The GCT model initializes its cache structure in GCTStream.__init__ within lingbot_map/models/gct_stream_window_v2.py (lines 22-28). Each transformer block maintains a dictionary kv_cache storing tensors shaped [B, H, T, D] (batch, heads, token-time, head-dim), enabling causal attention to past frames without recomputing embeddings.

Configurable Capacity Limits

The system enforces memory bounds through three critical parameters:

• kv_cache_sliding_window (default 64): Maximum number of recent non-scale frames retained in GPU memory
• kv_cache_scale_frames (default 8): Fixed scale frames providing bidirectional attention for depth estimation
• kv_cache_include_scale_frames: Boolean flag determining whether scale frames coexist with the sliding window or operate separately

These parameters are passed to the AggregatorStream and attention layers during model construction, establishing the hard upper bound of 72 total cached frames (64 + 8) regardless of video duration.

Sliding-Window Eviction for Long Sequences

When the number of processed frames exceeds the configured capacity, the GCT triggers _apply_kv_cache_eviction_causal in lingbot_map/layers/attention.py (lines 297-350) immediately before attention computation.

Pre-Attention Cache Trimming

The eviction logic executes the following steps:

if num_cached_frames > sliding_window_frames + scale_frames:
    evict_start = scale_frames
    evict_end = num_cached_frames - sliding_window_frames
    # Remove frames between evict_start and evict_end

    # Retain scale frames (oldest) and most recent sliding window frames

This procedure removes the oldest non-scale frames from the middle of the sequence, then reconstructs the cache by concatenating the retained scale frames with the newest sliding_window_frames entries. Because eviction occurs before attention matrix construction, the causal mask requires no recomputation.

Special Token Preservation

Even when evicting frame embeddings, the model preserves global context through kv_cache_cross_frame_special. When enabled, special tokens (camera tokens, register tokens, and scale tokens) from evicted frames transfer to auxiliary buffers named k_{i}_special and v_{i}_special (lines 18-28 in attention.py). This ensures camera pose information and other cross-frame metadata remain accessible for downstream heads despite the sliding window constraint.

Key-Frame Sparsity and Memory Optimization

To further reduce memory footprint during long sequences, the GCT implements selective KV storage via the _set_skip_append(True) method (lines 14-22 in gct_stream_window_v2.py).

Skip-Append Mechanism

During inference_streaming, the model evaluates each frame against keyframe_interval (default 4) and flow-based thresholds:

• Keyframes: Full KV tensors are computed, stored, and subject to sliding-window eviction
• Non-keyframes: The model sets _skip_append before storage, using the KV for immediate attention but discarding it afterward

With a keyframe interval of 4, effective cache size drops to approximately 18 frames (72 / 4), enabling processing of hour-long videos with minimal GPU memory.

Rollback and Defer-Eviction

For frames initially classified as non-keyframes but later promoted based on optical flow thresholds, the model supports cache rollback and deferred eviction (lines 42-48 in gct_stream_window_v2.py). This mechanism removes the previous append, executes the deferred eviction routine, and repopulates the cache with corrected keyframe data, ensuring temporal consistency without memory duplication.

Practical Implementation

Creating a GCT model with optimized long-sequence parameters:

from lingbot_map.models.gct_stream_window_v2 import GCTStream

model = GCTStream(
    img_size=518,
    patch_size=14,
    embed_dim=1024,
    enable_stream_inference=True,
    kv_cache_sliding_window=64,      # Keep at most 64 recent frames

    kv_cache_scale_frames=8,         # Always retain 8 scale frames

    kv_cache_include_scale_frames=True,
    keyframe_interval=4,             # Only every 4th frame stores KV

)

Processing video streams exceeding 320 frames:

# video tensor shape: [N, 3, H, W] where N > 320

model.clean_kv_cache()  # Reset buffers (lines 88-97)

predictions = model.inference_streaming(
    images=video,
    keyframe_interval=4,
    output_device='cpu'
)

Monitoring cache statistics:

info = model.get_kv_cache_info()
print(info)  # Output: {'num_cached_blocks': 12, 'cache_memory_mb': 14.5}

The get_kv_cache_info() method (lines 84-92) reports current tensor sizes and approximate memory footprints, useful for debugging memory usage during deployment.

Summary

• Hard memory cap: The GCT KV cache enforces a strict limit of kv_cache_sliding_window + kv_cache_scale_frames frames (default 72), preventing out-of-memory errors on arbitrarily long videos
• Eviction timing: _apply_kv_cache_eviction_causal executes before attention in lingbot_map/layers/attention.py, removing oldest non-scale frames while preserving geometric consistency
• Key-frame sparsity: The _set_skip_append mechanism reduces effective cache entries by 75% when using default intervals, processing thousands of frames with constant memory
• Cross-frame persistence: Special tokens survive eviction via dedicated *_special buffers, maintaining global camera pose information across the entire sequence
• Rollback capability: Deferred eviction and rollback support correct keyframe promotion without cache corruption

Frequently Asked Questions

What happens when the GCT KV cache reaches 320 frames?

The cache never accumulates 320 frames. At 72 frames (default), the sliding-window eviction automatically triggers, discarding the oldest non-scale frames while retaining the most recent 64 frames plus 8 fixed scale frames. This circular buffer behavior ensures constant memory usage regardless of video length.

How does the sliding window affect temporal attention quality?

The model maintains causal attention to the most recent 64 frames plus 8 scale frames, providing approximately 2-3 seconds of temporal context at standard frame rates. This window captures sufficient motion history for depth estimation and camera pose prediction while preventing the quadratic memory growth typical of full-sequence attention.

Can I adjust the KV cache size for longer temporal windows?

Yes, increase kv_cache_sliding_window when initializing GCTStream, though GPU memory consumption scales linearly with the window size. For extremely long dependencies, combine larger windows with decreased keyframe_interval to balance memory and temporal reach.

What are scale frames and why are they exempt from eviction?

Scale frames provide bidirectional attention necessary for metric depth estimation and geometric consistency. They remain fixed at the beginning of the cache (indices 0-7) to anchor the global coordinate system, ensuring that even as the sliding window advances, the model maintains absolute scale references for accurate 3D reconstruction.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}