How KV Cache Pages Are Managed and Recycled in LingBot-Map

LingBot-Map manages KV cache pages through a sliding-window eviction system that allocates live pages for active frames, recycles evicted pages into a free pool, and supports deferred eviction with rollback capabilities for non-keyframes.

LingBot-Map is an open-source streaming vision transformer that implements causal attention over video frames by storing key/value tensors in a GPU-resident KV cache. According to the Robbyant/lingbot-map repository source code, the system employs a sophisticated page-based memory management strategy that supports both FlashInfer and PyTorch SDPA backends while enabling efficient recycling of cache pages through sliding-window eviction and deferred rollback mechanisms.

Dual Backend Architecture

LingBot-Map supports two distinct backends for KV cache storage, each with different page management implementations.

FlashInfer Backend utilizes a C++ based kv_cache_manager that maintains distinct page pools: live pages for active frames, free pages for recycled memory, and special pages for tokens that must survive eviction. This backend handles eviction through native sliding-window logic implemented in the FlashInfer library.

SDPA (PyTorch) Backend uses a Python dict named kv_cache where tensors are stored as plain entries (k_0, v_0, etc.). Page handling is emulated through boolean flags including _skip_append and _defer_eviction that control whether frames are written to or evicted from the cache.

Page Allocation and Statistics

When processing a new frame, the system appends KV pairs to live pages or triggers eviction when the sliding-window limit is reached. The FlashInfer manager exposes detailed statistics through the get_cache_stats method.

In lingbot_map/models/gct_stream_window_v2.py (lines 65-71), the implementation retrieves per-block statistics:

mgr = getattr(agg, "kv_cache_manager", None)
if mgr is not None and hasattr(mgr, "get_cache_stats"):
    s = mgr.get_cache_stats(block_idx=0)
    parts.append(
        f"FI[blk0] frames={s['frame_count']} "
        f"scale_pg={s['scale_pages']} live_pg={s['live_pages']} "
        f"free_pg={s['free_pages']} special={s['special_tokens']}"
    )

These statistics track live pages (currently active), free pages (available for reuse), scale pages (multi-resolution frames), and special tokens (protected from eviction).

Sliding-Window Eviction Configuration

The cache size is controlled by two primary parameters initialized in the model's __init__ method (lines 222-229 of gct_stream_window_v2.py):

self.kv_cache_sliding_window = kv_cache_sliding_window
self.kv_cache_scale_frames = kv_cache_scale_frames

The kv_cache_sliding_window parameter (default 64) defines how many keyframes are retained before eviction occurs. When the frame count exceeds this window, the oldest pages are moved to the free pool for recycling.

Executing Deferred Eviction

Eviction is performed through the execute_deferred_eviction method after a keyframe is confirmed. This logic appears in lines 75-82 of gct_stream_window_v2.py:

mgr.execute_deferred_eviction(
    block_idx,
    scale_frames=self.kv_cache_scale_frames,
    sliding_window=self.kv_cache_sliding_window,
)

This method calculates which pages fall outside the sliding window and transfers them from the live pool to the free pool, making them available for subsequent frame allocations without requiring new GPU memory allocation.

Deferred Eviction and Rollback for Non-Keyframes

For frames that are not keyframes, the system implements a deferred eviction pattern to enable rollback if the frame is rejected.

First, eviction is suppressed using _set_defer_eviction(True) before the forward pass (lines 26-34). After processing, if the frame is not selected as a keyframe, the system rolls back the cache modifications:

self._set_defer_eviction(True)   # suppress eviction

frame_output = self.forward(...)
self._set_defer_eviction(False)

if not is_keyframe:
    self._rollback_last_frame()

The rollback implementation (lines 49-55 and 68-71) handles both backends:

mgr.rollback_last_frame(block_idx)                # FlashInfer path

kv[key] = kv[key][:, :, :-1]                     # SDPA dict path

self.aggregator.total_frames_processed -= 1

This mechanism ensures that tentative frames do not permanently consume cache pages or trigger premature eviction of valid keyframes.

Page Behavior Control Flags

Several flags control how pages are allocated and evicted:

• _skip_append: Prevents storing the current frame's KV tensors, used for non-keyframes. Set via _set_skip_append (lines 14-21).
• _defer_eviction: Suppresses eviction until the frame is confirmed. Set via _set_defer_eviction (lines 26-34).
• kv_cache_cross_frame_special: Preserves special tokens (e.g., CLS) from evicted frames across the sequence.
• kv_cache_include_scale_frames: Includes scale-frame KV tensors in the sliding window calculation.
• kv_cache_camera_only: Retains only camera tokens during eviction for camera-head specific pipelines.

These flags are passed to the AggregatorStream during initialization (lines 27-31) and stored in the cache container for consultation during eviction logic.

Cache Cleanup and Memory Management

Between video sequences, the entire cache is cleared via the clean_kv_cache method (lines 88-99):

def clean_kv_cache(self):
    if hasattr(self.aggregator, 'clean_kv_cache'):
        self.aggregator.clean_kv_cache()
    if hasattr(self.camera_head, 'kv_cache'):
        self.camera_head.clean_kv_cache()

Memory utilization is estimated based on tensor element counts assuming bfloat16 storage (2 bytes per element), as shown in lines 100-106:

total_elements = sum(v.numel() for v in kv_cache.values() if v is not None)
cache_memory_mb = (total_elements * 2) / (1024 * 1024)

Practical Implementation Examples

Inspecting Cache Statistics During Streaming

Monitor page allocation in real-time to debug memory usage:

model = GCTStream(..., kv_cache_sliding_window=32, kv_cache_scale_frames=4)
for frame in video:
    out = model.forward(frame, causal_inference=True)
    if frame_idx % 10 == 0:
        model._log_kv_stats(label=f"frame {frame_idx}")

Forcing Cache Reset Between Sequences

Prevent cross-sequence contamination when processing multiple videos:

model.clean_kv_cache()          # Clears aggregator & camera head caches

pred1 = model.inference_streaming(clip1)
model.clean_kv_cache()          # Reset before next clip

pred2 = model.inference_streaming(clip2)

Limiting Memory with Small Sliding Windows

Reduce GPU memory footprint for long videos:

model = GCTStream(
    enable_stream_inference=True,
    kv_cache_sliding_window=16,   # keep only the last 16 keyframes

    kv_cache_scale_frames=2,
)
pred = model.inference_streaming(video, keyframe_interval=2)

Summary

• LingBot-Map implements a dual-backend KV cache supporting both FlashInfer (C++ manager) and SDPA (Python dict) implementations.
• Page recycling operates through a sliding-window mechanism where evicted pages move to a free pool rather than being deallocated.
• Deferred eviction allows tentative processing of non-keyframes with full rollback capability via rollback_last_frame.
• Control flags (_skip_append, _defer_eviction, and configuration options) provide fine-grained control over which tokens are cached and evicted.
• Memory estimation is based on bfloat16 tensor sizes, with the sliding window and scale frame settings directly determining maximum GPU RAM usage.

Frequently Asked Questions

How does LingBot-Map decide which KV cache pages to evict?

LingBot-Map uses a sliding-window algorithm controlled by the kv_cache_sliding_window parameter. When the number of stored keyframes exceeds this limit, the oldest pages are moved from the live pool to the free pool via the execute_deferred_eviction method. Special tokens marked by kv_cache_cross_frame_special are preserved in a separate pool and are not subject to standard eviction.

What happens to KV cache pages when a frame is not selected as a keyframe?

When processing non-keyframes, the system sets _defer_eviction(True) to prevent immediate eviction, runs the forward pass, then calls _rollback_last_frame() if the frame is rejected. This rollback removes the temporarily appended KV entries and restores the frame counter, effectively recycling the allocated pages back to their pre-frame state without triggering the sliding-window eviction logic.

How can I monitor KV cache memory usage in real-time?

You can access the get_cache_stats method on the FlashInfer manager to retrieve live counts of frames, scale pages, live pages, free pages, and special tokens. For SDPA backends, calculate memory usage by summing tensor elements and multiplying by 2 bytes (for bfloat16), then converting to megabytes using the formula (total_elements * 2) / (1024 * 1024).

What is the difference between FlashInfer and SDPA KV cache management?

FlashInfer uses a native C++ manager with explicit page pools (live, free, special) and built-in sliding-window eviction, while SDPA uses a Python dictionary where pages are managed as tensor slices with eviction logic emulated through boolean flags like _skip_append and _defer_eviction. FlashInfer generally provides more efficient memory recycling for production streaming, whereas SDPA offers simpler debugging and flexibility during development.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}