# What Are Scale Frames in LingBot-Map and How Do They Work During Inference?

> Understand LingBot-Map scale frames. Learn how they establish global metric scale for consistent 3D pose predictions during inference.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: deep-dive
- Published: 2026-06-29

---

**Scale frames in LingBot-Map are the initial batch of video frames processed together with bidirectional attention to establish a global metric scale, which is then cached and referenced throughout streaming or windowed inference to maintain consistent 3D pose predictions.**

In the Robbyant/lingbot-map repository, scale frames serve as the foundational anchor for the model's metric understanding of the scene. These frames enable the transformer to compute the conversion factor between image-space and world-space coordinates before processing the remainder of the video stream.

## What Are Scale Frames in LingBot-Map?

Scale frames are a short contiguous sequence of initial video frames that the model processes as a single block before beginning frame-by-frame streaming. Unlike the subsequent streaming phase where causal or windowed attention is used, these frames receive **bidirectional attention**, allowing the model to estimate a global *scale* factor.

This scale factor represents the metric conversion between pixels in the image and real-world distances. Without this initial calibration, the 3D pose predictions would drift or become inconsistent as the video progresses. The mechanism is implemented in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py), where the model uses a special *scale token* to facilitate this cross-frame attention among the initial batch.

## Configuration Parameters for Scale Frames

Several key parameters control how scale frames are handled during inference:

- **`num_frame_for_scale`**: Defines how many initial frames are treated as scale frames (commonly set to 4 or 8).
- **`attend_to_scale_frames`**: A boolean flag that determines whether subsequent streaming frames can attend to the scale frame representations.
- **`kv_cache_scale_frames`**: Controls how many scale frames remain resident in the KV cache (defaults to 8).

These parameters are initialized in the model constructor and can be overridden during inference calls.

## The Two-Phase Inference Architecture

The inference process explicitly separates scale frame processing from the streaming pipeline.

### Phase 1: Batch Processing with Bidirectional Attention

When `inference_streaming` or `inference_windowed` is invoked, the model first slices out the initial `num_frame_for_scale` frames. As seen in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py) (lines 524-528), these frames are moved to the device and fed to the transformer as a single block using `num_frame_per_block=scale_frames`.

This triggers bidirectional attention among the scale frames via the scale token, computing the metric scale without the causal masking used in the streaming phase. The KV cache is populated with these representations, creating a permanent reference point for the sequence.

### Phase 2: Streaming with Cached Scale Context

After the initial batch processing, the remaining frames are processed one-by-one in streaming mode. The model retains the KV cache from Phase 1, allowing each new frame to attend to the scale frame context without recomputing the initial representations.

If `attend_to_scale_frames` is enabled, the attention mechanism explicitly queries the scale frame entries in the KV cache (lines 512-514). This ensures that the metric scale established during Phase 1 remains available throughout the entire sequence, maintaining consistency in 3D pose predictions.

## Scale Frames in Windowed Inference

In `inference_windowed`, scale frames serve a dual purpose as the overlapping region between consecutive windows. According to the implementation in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py) (lines 1044-1051), the overlap size defaults to the number of scale frames, though it can be overridden using `overlap_keyframes` or `overlap_size`.

This overlap ensures that each window maintains alignment with the global metric scale, preventing discontinuities at window boundaries. The scale frames act as a reliable reference that bridges separate processing windows.

## KV Cache Management and Memory Optimization

Scale frames receive special treatment in the KV cache hierarchy. By default, `kv_cache_include_scale_frames` is set to `True`, ensuring these frames remain resident in cache regardless of keyframe eviction policies.

The [`flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/flashinfer_cache.py) module defines the default number of always-resident scale frames as 8, controlled by the `kv_cache_scale_frames` parameter (lines 222-227 in [`gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream_window_v2.py)). This reserved cache space guarantees that the metric reference remains available for any future keyframe or non-keyframe attention, even when the cache is under pressure from long sequences.

## Practical Implementation Example

The following example demonstrates how to configure and utilize scale frames in LingBot-Map:

```python
import torch
from lingbot_map.models.gct_stream_window_v2 import GCTStream

# Initialize model with scale frame parameters

model = GCTStream(
    img_size=518,
    patch_size=14,
    embed_dim=1024,
    patch_embed='dinov2_vitl14_reg',
    enable_camera=True,
    enable_point=False,
    enable_depth=False,
    num_frame_for_scale=4,          # Use 4 scale frames

    attend_to_scale_frames=True,    # Enable attention to scale frames

)

# Dummy video: 12 RGB frames, 3×480×640

videos = torch.randn(1, 12, 3, 480, 640)

# Streaming inference: first 4 frames processed together,

# remaining 8 frames processed one-by-one

preds = model.inference_streaming(
    images=videos,
    num_scale_frames=4,            # Override default if needed

    keyframe_interval=1,
)

print(preds['pose_enc'].shape)    # → (1, 12, 9)

```

For windowed inference with explicit overlap control:

```python

# Windowed inference with scale frames as overlap

windowed_preds = model.inference_windowed(
    images=videos,
    window_size=6,                 # 6 frames per window

    overlap_keyframes=2,           # Overlap expressed in keyframes

    num_scale_frames=4,
)

print(windowed_preds['pose_enc'].shape)  # → (1, 12, 9)

```

## Summary

- **Scale frames** are the initial batch of frames processed with bidirectional attention in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py) to establish global metric scale.
- The **two-phase inference** process separates scale estimation (Phase 1) from streaming prediction (Phase 2).
- **KV cache management** ensures scale frames remain resident via `kv_cache_scale_frames` (default 8) and `kv_cache_include_scale_frames`.
- In **windowed inference**, scale frames provide the overlapping region between windows to maintain alignment.
- **Configuration parameters** `num_frame_for_scale` and `attend_to_scale_frames` control the quantity and accessibility of scale frame context.

## Frequently Asked Questions

### How many scale frames should I use for optimal performance?

Most configurations use between 4 and 8 scale frames, with the default `kv_cache_scale_frames` set to 8 in the flashinfer cache implementation. The `num_frame_for_scale` parameter should match your cache allocation to ensure all scale frames remain resident during inference.

### Can I disable scale frames during inference?

While you can set `num_frame_for_scale=0`, this is not recommended as it removes the model's ability to establish a consistent metric scale between image and world coordinates. If memory is constrained, reduce the number rather than eliminating them entirely, and ensure `attend_to_scale_frames` reflects your attention requirements.

### How do scale frames affect memory usage in the KV cache?

Scale frames are always kept in the KV cache when `kv_cache_include_scale_frames=True`, consuming memory proportional to `kv_cache_scale_frames` multiplied by the attention head dimensions. This reserved space ensures the metric reference remains available even when the cache evicts other keyframes.

### What is the difference between scale frames and regular keyframes?

Scale frames are processed at the start with bidirectional attention to compute global scale, while regular keyframes are processed during streaming with causal or windowed attention. Scale frames remain permanently in the KV cache, whereas regular keyframes may be evicted based on the cache management policy and window size.