Understanding the `clean_kv_cache` Function in LingBot-Map: When to Manually Reset KV Caches

The clean_kv_cache function resets all cached key-value (KV) tensors in streaming models, aggregators, and camera heads to prevent stale attention across video sequences and avoid GPU memory exhaustion.

The LingBot-Map repository implements transformer-based video understanding with streaming inference, where the clean_kv_cache function plays a critical role in managing the lifecycle of attention caches. When processing sequential video frames, the model accumulates KV tensors in a paged cache to enable efficient causal attention, but this cache must be explicitly cleared between independent sequences to ensure correct predictions and prevent memory leaks.

What Does the clean_kv_cache Function Do?

The clean_kv_cache function is a utility method implemented across several core classes in the LingBot-Map architecture. Its primary responsibility is to reset all cached key-value tensors that accumulate during streaming inference runs, effectively returning the model to a pristine state before processing new input.

Role in the FlashInfer Architecture

Inside the model's transformer layers, the FlashInfer KV cache manager (FlashInferKVCacheManager) maintains a paged cache that stores the K and V tensors of previously processed frames. This enables causal attention across video frames without recomputing the entire sequence history. However, when transitioning to a new video sequence, the cache still holds tensors from the previous video.

If the cache is not cleared, the attention mechanism for the new video incorrectly attends to stale tokens from previous sequences, resulting in erroneous pose or depth predictions. Additionally, uncleared pages accumulate indefinitely, leading to GPU memory exhaustion.

Components That Implement Cache Clearing

The clean_kv_cache method exists on multiple high-level components:

• Streaming models: GCTStream, GCTStreamWindow, and GCTStreamWindowV2
• Aggregator: AggregatorStream
• Camera head: CameraCausalHead

In lingbot_map/models/gct_stream.py (lines 84-90), the implementation traverses the model hierarchy and delegates to subcomponents:

def clean_kv_cache(self):
    """Clean KV cache in aggregator."""
    if hasattr(self.aggregator, 'clean_kv_cache'):
        self.aggregator.clean_kv_cache()
    else:
        logger.warning("Aggregator does not support KV cache cleaning")
    if hasattr(self.camera_head, 'kv_cache'):
        self.camera_head.clean_kv_cache()
    else:
        logger.warning("Camera head does not support KV cache cleaning")

The underlying reset logic resides in lingbot_map/layers/flashinfer_cache.py (lines 23-33), where the FlashInferKVCacheManager.reset() method zeros all per-block deques and counters.

When to Call clean_kv_cache Manually

While the high-level API handles cache clearing automatically in most cases, several scenarios require explicit manual invocation of the clean_kv_cache function.

Switching Between Video Sequences

When processing a new video or image sequence, you must clear the KV cache to prevent cross-contamination between videos. If stale KV tensors persist, the model's attention mechanism references tokens from previous sequences, producing incorrect predictions.

Recommended call site: At the beginning of your inference loop before processing the first frame of a new video.

Running Multiple Independent Inference Passes

If you are running multiple independent inference passes within the same Python process, manual cache clearing is essential between runs. This prevents information leakage between unrelated videos when you want a fresh cache for each inference pass.

Recommended call site: Immediately before each model.forward() or model.inference_streaming() call if you bypass the high-level streaming API.

Memory Management and Debugging

After long streaming sessions, calling clean_kv_cache frees GPU memory that is no longer needed, preventing out-of-memory (OOM) errors during extended operation. During debugging or profiling, manual cache clearing allows you to isolate the effects of KV cache reuse and measure baseline performance without accumulated state.

Note: The inference_streaming method in GCTStream already invokes self.clean_kv_cache() automatically after moving input to the device (see line 106). You only need manual calls when using low-level APIs or when resetting the cache mid-stream (for example, after a deliberate scene change).

Code Implementation and File Locations

The cache clearing mechanism spans four critical files in the repository:

File	Role
lingbot_map/models/gct_stream.py	Defines GCTStream.clean_kv_cache and invokes it in inference_streaming
lingbot_map/aggregator/stream.py	Implements AggregatorStream.clean_kv_cache, calling the FlashInfer manager
lingbot_map/heads/camera_head.py	Provides CameraCausalHead.clean_kv_cache for per-camera caches
lingbot_map/layers/flashinfer_cache.py	Core FlashInfer KV cache manager; reset() empties internal deques

The FlashInferKVCacheManager.reset() method (lines 23-33) performs the actual memory reset by clearing the underlying data structures used for paged attention.

Practical Usage Examples

Example 1: High-Level API Usage

When using inference_streaming, the cache clears automatically:

model = GCTStream(...)
predictions = model.inference_streaming(video_tensor)   # clean_kv_cache() called internally

Example 2: Low-Level Forward Calls

When using forward() directly, you must manually manage the cache lifecycle:

model = GCTStream(...)
model.clean_kv_cache()          # explicit reset before first frame

for frame in video_frames:
    out = model.forward(frame)  # KV cache grows as frames process

# Optional reset after sequence

model.clean_kv_cache()

Example 3: Handling Scene Changes

Reset the cache mid-stream when processing distinct segments:

for segment in segments:
    model.clean_kv_cache()          # clear before new segment

    seg_pred = model.inference_streaming(segment)
    # process predictions...

Summary

• The clean_kv_cache function resets KV tensors in GCTStream, AggregatorStream, and CameraCausalHead to prevent attention contamination.
• Call it manually when switching videos, running multiple independent inferences, or debugging to ensure state isolation.
• The high-level inference_streaming method automatically clears the cache at line 106, but low-level forward() usage requires explicit management.
• The underlying implementation resides in FlashInferKVCacheManager.reset() within lingbot_map/layers/flashinfer_cache.py.

Frequently Asked Questions

What happens if I don't call clean_kv_cache between videos?

The model's attention mechanism will attend to stale KV tensors from previous sequences, causing incorrect pose and depth predictions. Additionally, GPU memory will accumulate uncleared pages, eventually causing out-of-memory errors.

Does inference_streaming call clean_kv_cache automatically?

Yes. The inference_streaming method in lingbot_map/models/gct_stream.py invokes self.clean_kv_cache() automatically after moving input to the device (around line 106). You only need manual calls when bypassing this high-level API or resetting mid-stream.

Which components have their own clean_kv_cache methods?

The method is implemented on three architectural levels: the streaming models (GCTStream and variants), the aggregator (AggregatorStream), and the camera head (CameraCausalHead). The high-level model method delegates to these subcomponents to ensure complete cache clearance.

How does clean_kv_cache relate to FlashInfer?

The function serves as a wrapper around FlashInferKVCacheManager.reset(). While clean_kv_cache handles the high-level orchestration across model components, the actual memory reset occurs in lingbot_map/layers/flashinfer_cache.py where the paged attention buffers are zeroed and deques are cleared.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}