Scale Frames in LingBot-Map's KV Cache: Bidirectional Initialization for Streaming Video

Scale frames in LingBot-Map are the first N frames of a video sequence processed together as a batch to initialize the KV cache with bidirectional context, enabling accurate metric-scale estimation before switching to causal frame-by-frame inference.

In the Robbyant/lingbot-map repository, scale frames solve the cold-start problem in streaming video understanding. By processing an initial subset of frames together rather than sequentially, the model establishes a stable reference for metric conversion from pixel-space to world-space. This initialization phase populates the FlashInfer/SDPA cache with critical context that subsequent frames attend to throughout the sequence.

What Are Scale Frames?

Scale frames are a specialized subset of video frames defined by the kv_cache_scale_frames parameter (default 8). Unlike the remainder of the sequence, these frames are processed bidirectionally as a batch during the initialization phase.

The primary purpose is to provide a short, bidirectional context window that allows the network to estimate scene-scale—the metric conversion from pixel-space to world-space—before streaming inference begins. When kv_cache_include_scale_frames=True, the key-value (KV) entries generated during this phase persist in the cache, allowing later frames to attend to them without recomputation.

How Scale Frames Work in the KV Cache

The implementation follows a two-phase architecture in gct_stream_window_v2.py:

Phase 1: Scale Frame Processing

During the scale phase, the model extracts and processes all scale frames together in a single forward pass. According to the source code in inference_streaming (lines 90-98), the flow is:

scale_images = images[:, :scale_frames]          # line 93-94

scale_output = self.forward(                     # line 95-99

    scale_images,
    num_frame_for_scale=scale_frames,
    num_frame_per_block=scale_frames,
    causal_inference=True,
)

This batch processing occurs before any frame-by-frame streaming begins. The KV entries produced here remain in the cache when kv_cache_include_scale_frames=True, establishing a permanent reference point.

Phase 2: Frame-by-Frame Streaming

After the scale phase completes, the model switches to causal inference. Each subsequent frame is processed individually, with queries attending to the cached KVs of earlier frames including the scale frames. The cache management logic uses _set_skip_append and _set_defer_eviction to ensure these initial frames remain accessible throughout the sequence.

This design reduces memory footprint while maintaining access to the initial bidirectional context, as the scale frames act as a stable reference for the entire video sequence.

Configuring Scale Frames in LingBot-Map

You can customize scale frame behavior through constructor parameters defined at lines 224-230 of lingbot_map/models/gct_stream_window_v2.py.

Customizing the Number of Scale Frames

from lingbot_map.models.gct_stream_window_v2 import GCTStream

model = GCTStream(
    img_size=518,
    patch_size=14,
    embed_dim=1024,
    kv_cache_scale_frames=5,          # use 5 scale frames instead of default 8

    kv_cache_include_scale_frames=True,
    enable_stream_inference=True,
)

Running Streaming Inference

The scale phase invokes automatically when calling inference_streaming:

# images: tensor of shape [B, S, 3, H, W]

pred = model.inference_streaming(
    images,
    num_scale_frames=5,      # optional override (uses self.kv_cache_scale_frames if omitted)

    keyframe_interval=2,      # every 2nd frame after scale phase is stored in KV cache

)

Source: inference_streaming implementation (lines 90-100) extracts scale_frames and processes them together before beginning the streaming phase.

Monitoring KV Cache Statistics

Check scale frame occupancy and cache utilization:

stats = model.get_kv_cache_info()
print(stats)   # {'num_cached_blocks': X, 'cache_memory_mb': Y}

Source: get_kv_cache_info reports cached block counts including scale frames when kv_cache_include_scale_frames is true (lines 84-92).

Key Source Files

Understanding scale frames requires examining these specific modules:

• lingbot_map/models/gct_stream_window_v2.py – Core model defining kv_cache_scale_frames, the scale-frame processing logic in inference_streaming, and KV-cache initialization.

• lingbot_map/aggregator/stream.py – Implements AggregatorStream that receives kv_cache_scale_frames and manages scale frame storage in the FlashInfer cache.

• lingbot_map/layers/flashinfer_cache.py – Low-level KV-cache manager (FlashInfer backend) holding cached K/V pairs from the scale phase.

• scripts/benchmark_gct_memory.py – Benchmark script exposing --num-scale-frames for tuning memory usage and performance.

Summary

• Scale frames are the first N frames (default 8) processed bidirectionally as a batch to initialize the KV cache with metric-scale reference data.
• The scale phase occurs in inference_streaming (lines 90-98) before causal streaming begins, implemented in gct_stream_window_v2.py.
• When kv_cache_include_scale_frames=True, scale frame KV entries persist in the FlashInfer cache, allowing subsequent frames to attend to them without recomputation.
• Configuration occurs via kv_cache_scale_frames and kv_cache_include_scale_frames parameters in the GCTStream constructor.
• Scale frames reduce memory footprint while maintaining access to critical bidirectional context throughout the video sequence.

Frequently Asked Questions

What is the default number of scale frames in LingBot-Map?

The default value is 8, controlled by the kv_cache_scale_frames parameter in the GCTStream constructor. You can override this when instantiating the model or when calling inference_streaming.

How do scale frames differ from keyframes in the KV cache?

Scale frames are processed once at the beginning of the sequence as a bidirectional batch to establish metric scale, while keyframes are selected frames stored throughout the sequence based on keyframe_interval. Scale frames provide initialization context; keyframes provide temporal landmarks during streaming.

Can I disable scale frames in streaming inference?

While you can set kv_cache_scale_frames to a lower value like 1, setting it to 0 would disable the scale phase entirely. However, this is not recommended as the model relies on the initial bidirectional context for accurate metric estimation.

Where is the scale frame logic implemented in the source code?

The primary implementation resides in lingbot_map/models/gct_stream_window_v2.py, specifically in the inference_streaming method (lines 90-100) and the constructor (lines 224-230). The FlashInfer backend storage is handled in lingbot_map/layers/flashinfer_cache.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}