# How KV Cache Pages Are Managed and Recycled in LingBot-Map

> Discover how LingBot-Map handles KV cache pages with its sliding-window eviction system. Learn about live page allocation, free pool recycling, and deferred eviction with rollback.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: internals
- Published: 2026-07-06

---

**LingBot-Map manages KV cache pages through a sliding-window eviction system that allocates live pages for active frames, recycles evicted pages into a free pool, and supports deferred eviction with rollback capabilities for non-keyframes.**

LingBot-Map is an open-source streaming vision transformer that implements causal attention over video frames by storing key/value tensors in a GPU-resident KV cache. According to the Robbyant/lingbot-map repository source code, the system employs a sophisticated page-based memory management strategy that supports both FlashInfer and PyTorch SDPA backends while enabling efficient recycling of cache pages through sliding-window eviction and deferred rollback mechanisms.

## Dual Backend Architecture

LingBot-Map supports two distinct backends for KV cache storage, each with different page management implementations.

**FlashInfer Backend** utilizes a C++ based `kv_cache_manager` that maintains distinct page pools: *live* pages for active frames, *free* pages for recycled memory, and *special* pages for tokens that must survive eviction. This backend handles eviction through native sliding-window logic implemented in the FlashInfer library.

**SDPA (PyTorch) Backend** uses a Python `dict` named `kv_cache` where tensors are stored as plain entries (`k_0`, `v_0`, etc.). Page handling is emulated through boolean flags including `_skip_append` and `_defer_eviction` that control whether frames are written to or evicted from the cache.

## Page Allocation and Statistics

When processing a new frame, the system appends KV pairs to live pages or triggers eviction when the sliding-window limit is reached. The FlashInfer manager exposes detailed statistics through the `get_cache_stats` method.

In [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py) (lines 65-71), the implementation retrieves per-block statistics:

```python
mgr = getattr(agg, "kv_cache_manager", None)
if mgr is not None and hasattr(mgr, "get_cache_stats"):
    s = mgr.get_cache_stats(block_idx=0)
    parts.append(
        f"FI[blk0] frames={s['frame_count']} "
        f"scale_pg={s['scale_pages']} live_pg={s['live_pages']} "
        f"free_pg={s['free_pages']} special={s['special_tokens']}"
    )

```

These statistics track **live pages** (currently active), **free pages** (available for reuse), **scale pages** (multi-resolution frames), and **special tokens** (protected from eviction).

## Sliding-Window Eviction Configuration

The cache size is controlled by two primary parameters initialized in the model's `__init__` method (lines 222-229 of [`gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream_window_v2.py)):

```python
self.kv_cache_sliding_window = kv_cache_sliding_window
self.kv_cache_scale_frames = kv_cache_scale_frames

```

The `kv_cache_sliding_window` parameter (default 64) defines how many keyframes are retained before eviction occurs. When the frame count exceeds this window, the oldest pages are moved to the free pool for recycling.

## Executing Deferred Eviction

Eviction is performed through the `execute_deferred_eviction` method after a keyframe is confirmed. This logic appears in lines 75-82 of [`gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream_window_v2.py):

```python
mgr.execute_deferred_eviction(
    block_idx,
    scale_frames=self.kv_cache_scale_frames,
    sliding_window=self.kv_cache_sliding_window,
)

```

This method calculates which pages fall outside the sliding window and transfers them from the live pool to the free pool, making them available for subsequent frame allocations without requiring new GPU memory allocation.

## Deferred Eviction and Rollback for Non-Keyframes

For frames that are not keyframes, the system implements a deferred eviction pattern to enable rollback if the frame is rejected.

First, eviction is suppressed using `_set_defer_eviction(True)` before the forward pass (lines 26-34). After processing, if the frame is not selected as a keyframe, the system rolls back the cache modifications:

```python
self._set_defer_eviction(True)   # suppress eviction

frame_output = self.forward(...)
self._set_defer_eviction(False)

if not is_keyframe:
    self._rollback_last_frame()

```

The rollback implementation (lines 49-55 and 68-71) handles both backends:

```python
mgr.rollback_last_frame(block_idx)                # FlashInfer path

kv[key] = kv[key][:, :, :-1]                     # SDPA dict path

self.aggregator.total_frames_processed -= 1

```

This mechanism ensures that tentative frames do not permanently consume cache pages or trigger premature eviction of valid keyframes.

## Page Behavior Control Flags

Several flags control how pages are allocated and evicted:

- **`_skip_append`**: Prevents storing the current frame's KV tensors, used for non-keyframes. Set via `_set_skip_append` (lines 14-21).
- **`_defer_eviction`**: Suppresses eviction until the frame is confirmed. Set via `_set_defer_eviction` (lines 26-34).
- **`kv_cache_cross_frame_special`**: Preserves special tokens (e.g., CLS) from evicted frames across the sequence.
- **`kv_cache_include_scale_frames`**: Includes scale-frame KV tensors in the sliding window calculation.
- **`kv_cache_camera_only`**: Retains only camera tokens during eviction for camera-head specific pipelines.

These flags are passed to the `AggregatorStream` during initialization (lines 27-31) and stored in the cache container for consultation during eviction logic.

## Cache Cleanup and Memory Management

Between video sequences, the entire cache is cleared via the `clean_kv_cache` method (lines 88-99):

```python
def clean_kv_cache(self):
    if hasattr(self.aggregator, 'clean_kv_cache'):
        self.aggregator.clean_kv_cache()
    if hasattr(self.camera_head, 'kv_cache'):
        self.camera_head.clean_kv_cache()

```

Memory utilization is estimated based on tensor element counts assuming bfloat16 storage (2 bytes per element), as shown in lines 100-106:

```python
total_elements = sum(v.numel() for v in kv_cache.values() if v is not None)
cache_memory_mb = (total_elements * 2) / (1024 * 1024)

```

## Practical Implementation Examples

### Inspecting Cache Statistics During Streaming

Monitor page allocation in real-time to debug memory usage:

```python
model = GCTStream(..., kv_cache_sliding_window=32, kv_cache_scale_frames=4)
for frame in video:
    out = model.forward(frame, causal_inference=True)
    if frame_idx % 10 == 0:
        model._log_kv_stats(label=f"frame {frame_idx}")

```

### Forcing Cache Reset Between Sequences

Prevent cross-sequence contamination when processing multiple videos:

```python
model.clean_kv_cache()          # Clears aggregator & camera head caches

pred1 = model.inference_streaming(clip1)
model.clean_kv_cache()          # Reset before next clip

pred2 = model.inference_streaming(clip2)

```

### Limiting Memory with Small Sliding Windows

Reduce GPU memory footprint for long videos:

```python
model = GCTStream(
    enable_stream_inference=True,
    kv_cache_sliding_window=16,   # keep only the last 16 keyframes

    kv_cache_scale_frames=2,
)
pred = model.inference_streaming(video, keyframe_interval=2)

```

## Summary

- **LingBot-Map** implements a dual-backend KV cache supporting both **FlashInfer** (C++ manager) and **SDPA** (Python dict) implementations.
- **Page recycling** operates through a sliding-window mechanism where evicted pages move to a free pool rather than being deallocated.
- **Deferred eviction** allows tentative processing of non-keyframes with full rollback capability via `rollback_last_frame`.
- **Control flags** (`_skip_append`, `_defer_eviction`, and configuration options) provide fine-grained control over which tokens are cached and evicted.
- **Memory estimation** is based on bfloat16 tensor sizes, with the sliding window and scale frame settings directly determining maximum GPU RAM usage.

## Frequently Asked Questions

### How does LingBot-Map decide which KV cache pages to evict?

LingBot-Map uses a sliding-window algorithm controlled by the `kv_cache_sliding_window` parameter. When the number of stored keyframes exceeds this limit, the oldest pages are moved from the live pool to the free pool via the `execute_deferred_eviction` method. Special tokens marked by `kv_cache_cross_frame_special` are preserved in a separate pool and are not subject to standard eviction.

### What happens to KV cache pages when a frame is not selected as a keyframe?

When processing non-keyframes, the system sets `_defer_eviction(True)` to prevent immediate eviction, runs the forward pass, then calls `_rollback_last_frame()` if the frame is rejected. This rollback removes the temporarily appended KV entries and restores the frame counter, effectively recycling the allocated pages back to their pre-frame state without triggering the sliding-window eviction logic.

### How can I monitor KV cache memory usage in real-time?

You can access the `get_cache_stats` method on the FlashInfer manager to retrieve live counts of frames, scale pages, live pages, free pages, and special tokens. For SDPA backends, calculate memory usage by summing tensor elements and multiplying by 2 bytes (for bfloat16), then converting to megabytes using the formula `(total_elements * 2) / (1024 * 1024)`.

### What is the difference between FlashInfer and SDPA KV cache management?

FlashInfer uses a native C++ manager with explicit page pools (live, free, special) and built-in sliding-window eviction, while SDPA uses a Python dictionary where pages are managed as tensor slices with eviction logic emulated through boolean flags like `_skip_append` and `_defer_eviction`. FlashInfer generally provides more efficient memory recycling for production streaming, whereas SDPA offers simpler debugging and flexibility during development.