# Scale Frames in LingBot-Map's KV Cache: Bidirectional Initialization for Streaming Video

> Discover scale frames in LingBot-Map KV cache. Learn how batch processing of initial frames provides bidirectional context for accurate metric-scale estimation before causal inference.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: deep-dive
- Published: 2026-07-03

---

**Scale frames in LingBot-Map are the first N frames of a video sequence processed together as a batch to initialize the KV cache with bidirectional context, enabling accurate metric-scale estimation before switching to causal frame-by-frame inference.**

In the Robbyant/lingbot-map repository, scale frames solve the cold-start problem in streaming video understanding. By processing an initial subset of frames together rather than sequentially, the model establishes a stable reference for metric conversion from pixel-space to world-space. This initialization phase populates the FlashInfer/SDPA cache with critical context that subsequent frames attend to throughout the sequence.

## What Are Scale Frames?

Scale frames are a specialized subset of video frames defined by the `kv_cache_scale_frames` parameter (default **8**). Unlike the remainder of the sequence, these frames are processed **bidirectionally** as a batch during the initialization phase.

The primary purpose is to provide a short, bidirectional context window that allows the network to estimate scene-scale—the metric conversion from pixel-space to world-space—before streaming inference begins. When `kv_cache_include_scale_frames=True`, the key-value (KV) entries generated during this phase persist in the cache, allowing later frames to attend to them without recomputation.

## How Scale Frames Work in the KV Cache

The implementation follows a two-phase architecture in [`gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream_window_v2.py):

### Phase 1: Scale Frame Processing

During the scale phase, the model extracts and processes all scale frames together in a single forward pass. According to the source code in `inference_streaming` (lines 90-98), the flow is:

```python
scale_images = images[:, :scale_frames]          # line 93-94

scale_output = self.forward(                     # line 95-99

    scale_images,
    num_frame_for_scale=scale_frames,
    num_frame_per_block=scale_frames,
    causal_inference=True,
)

```

This batch processing occurs before any frame-by-frame streaming begins. The KV entries produced here remain in the cache when `kv_cache_include_scale_frames=True`, establishing a permanent reference point.

### Phase 2: Frame-by-Frame Streaming

After the scale phase completes, the model switches to causal inference. Each subsequent frame is processed individually, with queries attending to the cached KVs of earlier frames **including** the scale frames. The cache management logic uses `_set_skip_append` and `_set_defer_eviction` to ensure these initial frames remain accessible throughout the sequence.

This design reduces memory footprint while maintaining access to the initial bidirectional context, as the scale frames act as a stable reference for the entire video sequence.

## Configuring Scale Frames in LingBot-Map

You can customize scale frame behavior through constructor parameters defined at lines 224-230 of [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py).

### Customizing the Number of Scale Frames

```python
from lingbot_map.models.gct_stream_window_v2 import GCTStream

model = GCTStream(
    img_size=518,
    patch_size=14,
    embed_dim=1024,
    kv_cache_scale_frames=5,          # use 5 scale frames instead of default 8

    kv_cache_include_scale_frames=True,
    enable_stream_inference=True,
)

```

### Running Streaming Inference

The scale phase invokes automatically when calling `inference_streaming`:

```python

# images: tensor of shape [B, S, 3, H, W]

pred = model.inference_streaming(
    images,
    num_scale_frames=5,      # optional override (uses self.kv_cache_scale_frames if omitted)

    keyframe_interval=2,      # every 2nd frame after scale phase is stored in KV cache

)

```

*Source:* `inference_streaming` implementation (lines 90-100) extracts `scale_frames` and processes them together before beginning the streaming phase.

### Monitoring KV Cache Statistics

Check scale frame occupancy and cache utilization:

```python
stats = model.get_kv_cache_info()
print(stats)   # {'num_cached_blocks': X, 'cache_memory_mb': Y}

```

*Source:* `get_kv_cache_info` reports cached block counts including scale frames when `kv_cache_include_scale_frames` is true (lines 84-92).

## Key Source Files

Understanding scale frames requires examining these specific modules:

- **[`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py)** – Core model defining `kv_cache_scale_frames`, the scale-frame processing logic in `inference_streaming`, and KV-cache initialization.

- **[`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py)** – Implements `AggregatorStream` that receives `kv_cache_scale_frames` and manages scale frame storage in the FlashInfer cache.

- **[`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py)** – Low-level KV-cache manager (FlashInfer backend) holding cached K/V pairs from the scale phase.

- **[`scripts/benchmark_gct_memory.py`](https://github.com/Robbyant/lingbot-map/blob/main/scripts/benchmark_gct_memory.py)** – Benchmark script exposing `--num-scale-frames` for tuning memory usage and performance.

## Summary

- **Scale frames** are the first N frames (default 8) processed bidirectionally as a batch to initialize the KV cache with metric-scale reference data.
- The **scale phase** occurs in `inference_streaming` (lines 90-98) before causal streaming begins, implemented in [`gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream_window_v2.py).
- When `kv_cache_include_scale_frames=True`, scale frame KV entries persist in the FlashInfer cache, allowing subsequent frames to attend to them without recomputation.
- Configuration occurs via `kv_cache_scale_frames` and `kv_cache_include_scale_frames` parameters in the `GCTStream` constructor.
- Scale frames reduce memory footprint while maintaining access to critical bidirectional context throughout the video sequence.

## Frequently Asked Questions

### What is the default number of scale frames in LingBot-Map?

The default value is **8**, controlled by the `kv_cache_scale_frames` parameter in the `GCTStream` constructor. You can override this when instantiating the model or when calling `inference_streaming`.

### How do scale frames differ from keyframes in the KV cache?

Scale frames are processed **once** at the beginning of the sequence as a bidirectional batch to establish metric scale, while keyframes are selected frames stored throughout the sequence based on `keyframe_interval`. Scale frames provide initialization context; keyframes provide temporal landmarks during streaming.

### Can I disable scale frames in streaming inference?

While you can set `kv_cache_scale_frames` to a lower value like 1, setting it to 0 would disable the scale phase entirely. However, this is not recommended as the model relies on the initial bidirectional context for accurate metric estimation.

### Where is the scale frame logic implemented in the source code?

The primary implementation resides in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py), specifically in the `inference_streaming` method (lines 90-100) and the constructor (lines 224-230). The FlashInfer backend storage is handled in [`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py).