What Are Scale Frames in LingBot-Map and How Do They Work During Inference?

Scale frames in LingBot-Map are the initial batch of video frames processed together with bidirectional attention to establish a global metric scale, which is then cached and referenced throughout streaming or windowed inference to maintain consistent 3D pose predictions.

In the Robbyant/lingbot-map repository, scale frames serve as the foundational anchor for the model's metric understanding of the scene. These frames enable the transformer to compute the conversion factor between image-space and world-space coordinates before processing the remainder of the video stream.

What Are Scale Frames in LingBot-Map?

Scale frames are a short contiguous sequence of initial video frames that the model processes as a single block before beginning frame-by-frame streaming. Unlike the subsequent streaming phase where causal or windowed attention is used, these frames receive bidirectional attention, allowing the model to estimate a global scale factor.

This scale factor represents the metric conversion between pixels in the image and real-world distances. Without this initial calibration, the 3D pose predictions would drift or become inconsistent as the video progresses. The mechanism is implemented in lingbot_map/models/gct_stream_window_v2.py, where the model uses a special scale token to facilitate this cross-frame attention among the initial batch.

Configuration Parameters for Scale Frames

Several key parameters control how scale frames are handled during inference:

• num_frame_for_scale: Defines how many initial frames are treated as scale frames (commonly set to 4 or 8).
• attend_to_scale_frames: A boolean flag that determines whether subsequent streaming frames can attend to the scale frame representations.
• kv_cache_scale_frames: Controls how many scale frames remain resident in the KV cache (defaults to 8).

These parameters are initialized in the model constructor and can be overridden during inference calls.

The Two-Phase Inference Architecture

The inference process explicitly separates scale frame processing from the streaming pipeline.

Phase 1: Batch Processing with Bidirectional Attention

When inference_streaming or inference_windowed is invoked, the model first slices out the initial num_frame_for_scale frames. As seen in lingbot_map/models/gct_stream_window_v2.py (lines 524-528), these frames are moved to the device and fed to the transformer as a single block using num_frame_per_block=scale_frames.

This triggers bidirectional attention among the scale frames via the scale token, computing the metric scale without the causal masking used in the streaming phase. The KV cache is populated with these representations, creating a permanent reference point for the sequence.

Phase 2: Streaming with Cached Scale Context

After the initial batch processing, the remaining frames are processed one-by-one in streaming mode. The model retains the KV cache from Phase 1, allowing each new frame to attend to the scale frame context without recomputing the initial representations.

If attend_to_scale_frames is enabled, the attention mechanism explicitly queries the scale frame entries in the KV cache (lines 512-514). This ensures that the metric scale established during Phase 1 remains available throughout the entire sequence, maintaining consistency in 3D pose predictions.

Scale Frames in Windowed Inference

In inference_windowed, scale frames serve a dual purpose as the overlapping region between consecutive windows. According to the implementation in lingbot_map/models/gct_stream_window_v2.py (lines 1044-1051), the overlap size defaults to the number of scale frames, though it can be overridden using overlap_keyframes or overlap_size.

This overlap ensures that each window maintains alignment with the global metric scale, preventing discontinuities at window boundaries. The scale frames act as a reliable reference that bridges separate processing windows.

KV Cache Management and Memory Optimization

Scale frames receive special treatment in the KV cache hierarchy. By default, kv_cache_include_scale_frames is set to True, ensuring these frames remain resident in cache regardless of keyframe eviction policies.

The flashinfer_cache.py module defines the default number of always-resident scale frames as 8, controlled by the kv_cache_scale_frames parameter (lines 222-227 in gct_stream_window_v2.py). This reserved cache space guarantees that the metric reference remains available for any future keyframe or non-keyframe attention, even when the cache is under pressure from long sequences.

Practical Implementation Example

The following example demonstrates how to configure and utilize scale frames in LingBot-Map:

import torch
from lingbot_map.models.gct_stream_window_v2 import GCTStream

# Initialize model with scale frame parameters

model = GCTStream(
    img_size=518,
    patch_size=14,
    embed_dim=1024,
    patch_embed='dinov2_vitl14_reg',
    enable_camera=True,
    enable_point=False,
    enable_depth=False,
    num_frame_for_scale=4,          # Use 4 scale frames

    attend_to_scale_frames=True,    # Enable attention to scale frames

)

# Dummy video: 12 RGB frames, 3×480×640

videos = torch.randn(1, 12, 3, 480, 640)

# Streaming inference: first 4 frames processed together,

# remaining 8 frames processed one-by-one

preds = model.inference_streaming(
    images=videos,
    num_scale_frames=4,            # Override default if needed

    keyframe_interval=1,
)

print(preds['pose_enc'].shape)    # → (1, 12, 9)

For windowed inference with explicit overlap control:

# Windowed inference with scale frames as overlap

windowed_preds = model.inference_windowed(
    images=videos,
    window_size=6,                 # 6 frames per window

    overlap_keyframes=2,           # Overlap expressed in keyframes

    num_scale_frames=4,
)

print(windowed_preds['pose_enc'].shape)  # → (1, 12, 9)

Summary

• Scale frames are the initial batch of frames processed with bidirectional attention in lingbot_map/models/gct_stream_window_v2.py to establish global metric scale.
• The two-phase inference process separates scale estimation (Phase 1) from streaming prediction (Phase 2).
• KV cache management ensures scale frames remain resident via kv_cache_scale_frames (default 8) and kv_cache_include_scale_frames.
• In windowed inference, scale frames provide the overlapping region between windows to maintain alignment.
• Configuration parameters num_frame_for_scale and attend_to_scale_frames control the quantity and accessibility of scale frame context.

Frequently Asked Questions

How many scale frames should I use for optimal performance?

Most configurations use between 4 and 8 scale frames, with the default kv_cache_scale_frames set to 8 in the flashinfer cache implementation. The num_frame_for_scale parameter should match your cache allocation to ensure all scale frames remain resident during inference.

Can I disable scale frames during inference?

While you can set num_frame_for_scale=0, this is not recommended as it removes the model's ability to establish a consistent metric scale between image and world coordinates. If memory is constrained, reduce the number rather than eliminating them entirely, and ensure attend_to_scale_frames reflects your attention requirements.

How do scale frames affect memory usage in the KV cache?

Scale frames are always kept in the KV cache when kv_cache_include_scale_frames=True, consuming memory proportional to kv_cache_scale_frames multiplied by the attention head dimensions. This reserved space ensures the metric reference remains available even when the cache evicts other keyframes.

What is the difference between scale frames and regular keyframes?

Scale frames are processed at the start with bidirectional attention to compute global scale, while regular keyframes are processed during streaming with causal or windowed attention. Scale frames remain permanently in the KV cache, whereas regular keyframes may be evicted based on the cache management policy and window size.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}