KV Cache Parameters in LingBot-Map: Complete Configuration Guide

LingBot-Map exposes five configurable KV cache parameters—kv_cache_sliding_window, kv_cache_scale_frames, kv_cache_cross_frame_special, kv_cache_include_scale_frames, and kv_cache_camera_only—that control sliding window eviction, persistent scale frames, and selective token retention to optimize memory usage during streaming video processing.

The Robbyant/lingbot-map repository implements a sophisticated key-value caching mechanism for attention states across video frames. These KV cache parameters allow developers to tune the trade-off between GPU memory consumption and temporal context retention in the GCTStreamWindowV2 model architecture.

KV Cache Parameter Reference

The following hyper-parameters define cache behavior in lingbot_map/models/gct_stream_window_v2.py (lines 223-260) and the sibling gct_stream_window.py implementation:

• kv_cache_sliding_window (int, default: 64): Sets the size of the sliding window in frames for KV-cache eviction. The cache retains only the most recent N frames and discards older entries to bound memory usage.

• kv_cache_scale_frames (int, default: 8): Specifies the number of scale frames to retain permanently in the KV cache, regardless of sliding-window eviction policies. These frames provide stable reference points across the video stream.

• kv_cache_cross_frame_special (bool, default: True): When enabled, the cache preserves special tokens (such as CLS tokens) from evicted frames, allowing later frames to attend to these semantic anchors even after their source frames leave the sliding window.

• kv_cache_include_scale_frames (bool, default: True): Determines whether scale frames are stored within the KV cache itself in addition to being retained for other computational purposes.

• kv_cache_camera_only (bool, default: False): When set to True, only camera-related tokens from evicted frames are retained, significantly reducing memory footprint for non-camera tokens in SLAM-heavy workloads.

Implementation in GCTStreamWindowV2

According to the source code in gct_stream_window_v2.py, these parameters are passed directly to the model constructor and stored as instance attributes. The model also exposes clean_kv_cache() to purge cached states and internal flags (_skip_append, _defer_eviction) that the inference engine toggles based on these settings.

# Example: creating a model with a custom KV-cache configuration

from lingbot_map.models.gct_stream_window_v2 import GCTStreamWindowV2

model = GCTStreamWindowV2(
    kv_cache_sliding_window=32,          # keep only the last 32 frames

    kv_cache_scale_frames=4,             # retain 4 scale frames

    kv_cache_cross_frame_special=True,  # keep special tokens on eviction

    kv_cache_include_scale_frames=True, # store scale frames in the KV-cache

    kv_cache_camera_only=False,          # store all token types

)

Runtime Cache Management

During inference, you can adjust parameters dynamically or explicitly clear the cache to manage memory spikes:

# Example: adjusting KV-cache parameters on the fly

def set_cache_parameters(model, window, scale):
    model.kv_cache_sliding_window = window
    model.kv_cache_scale_frames = scale

# Tighten memory budget after initialization

set_cache_parameters(model, window=16, scale=2)

# Explicitly purge the cache when switching video segments

model.clean_kv_cache()

Key Source Files and Architecture

The KV cache implementation spans several critical files in the repository:

• lingbot_map/models/gct_stream_window_v2.py: Primary definition of the KV-cache parameters and eviction logic (lines 223-260).

• lingbot_map/models/gct_stream_window.py: Legacy stream-window model implementing the same interface (lines 160-197).

• lingbot_map/layers/flashinfer_cache.py: Low-level cache implementation that the high-level KV cache hooks into for optimized memory access.

• scripts/benchmark_gct_memory.py: Demonstrates command-line parameter passing and memory profiling (lines 152-155).

Summary

• Five parameters control LingBot-Map’s KV cache: sliding window size, scale frame count, special token retention, scale frame inclusion, and camera-only mode.
• Default configuration balances context length (64 frames) with memory efficiency (8 scale frames).
• Dynamic adjustment is supported via direct attribute modification during runtime.
• Cache clearing is available through the clean_kv_cache() method.
• Source files are located in gct_stream_window_v2.py and flashinfer_cache.py.

Frequently Asked Questions

What is the default sliding window size for the KV cache?

The default kv_cache_sliding_window is set to 64 frames. This means the model maintains attention states for the 64 most recent video frames before evicting older entries to prevent unbounded memory growth.

How do I clear the KV cache during inference?

Call the clean_kv_cache() method on your model instance. This purges all cached key-value tensors and resets internal flags, which is useful when switching between unrelated video sequences or when you detect memory pressure during long streaming sessions.

What is the difference between scale frames and sliding window frames?

Scale frames (kv_cache_scale_frames) are preserved permanently as reference anchors across the entire video stream, while sliding window frames are subject to FIFO eviction once the window limit is reached. Scale frames provide stable temporal context, whereas sliding window frames provide recent local context.

When should I enable kv_cache_camera_only?

Enable kv_cache_camera_only=True when running SLAM or camera-pose estimation tasks where only camera-related tokens contribute meaningful information. This setting filters out non-camera tokens during eviction, reducing memory usage by up to 50% in scenes with heavy semantic segmentation overhead.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}