How to Calculate Actual Frame Coverage from Window Size in LingBot-Map

In LingBot-Map, the window_size parameter counts keyframes rather than raw video frames, so you must apply the formula actual_frames = scale_frames + (window_size - scale_frames) * keyframe_interval to determine the true temporal coverage of each processing window.

When configuring streaming inference in the Robbyant/lingbot-map repository, developers often confuse the window_size argument with the number of actual video frames processed. Since the window operates on keyframes—frames whose KV cache is retained after processing—the actual temporal coverage depends on the relationship between scale frames, keyframe intervals, and the window size. Understanding this calculation is essential for accurately controlling memory consumption and temporal context in GCTStream.inference_windowed.

Understanding Keyframes and Window Abstraction

In LingBot-Map, the window abstraction works on keyframes, which are specific frames whose key-value (KV) cache is retained after processing. The window_size argument passed to GCTStream.inference_windowed therefore counts keyframes, not the raw video frames.

A window contains two distinct processing phases:

• Scale frames: The first num_scale_frames frames processed together with bidirectional attention. All these frames are retained in the KV cache.
• Phase-2 frames: The remaining frames in the window, processed one-by-one with the KV cache. When keyframe_interval > 1, only every N-th Phase-2 frame becomes a keyframe; intervening frames are processed but their KV entries are discarded.

Consequently, a single window spans more actual video frames than the number of keyframes it contains when keyframe_interval is greater than 1.

The Frame Coverage Formula

The exact relationship between window size and actual frame coverage is documented in the source code at lingbot_map/models/gct_stream_window_v2.py (lines 1043-1048):

# ----------------------------------------------------------------------

# ``window_size`` counts **keyframes** (frames stored in KV cache),

# including scale frames.  When ``keyframe_interval > 1``, each window

# covers more actual frames than ``window_size``:

#

#     actual_frames = scale_frames + (window_size - scale_frames) * keyframe_interval

# ----------------------------------------------------------------------

Breaking down the formula:

• scale_frames (or num_scale_frames): The number of scale frames processed with bidirectional attention (default equals self.num_frame_for_scale).
• window_size: The total number of keyframes the window should contain, including the scale frames.
• keyframe_interval: The interval between consecutive Phase-2 keyframes (default = 1).

The derivation works as follows:

1. Scale phase: scale_frames (all kept in KV cache).
2. Phase-2 keyframes: window_size - scale_frames.
3. Phase-2 non-keyframes: (window_size - scale_frames) * (keyframe_interval - 1).

Summing these components yields the total actual frames: actual_frames = scale_frames + (window_size - scale_frames) * keyframe_interval.

When keyframe_interval == 1, the formula collapses to actual_frames == window_size, meaning every frame is a keyframe.

Practical Implementation

When configuring windowed inference, you typically need to translate between window size and actual coverage. Below are two utility functions that implement the core calculations.

Calculating Actual Frames from Window Size

Use this function when you know the window configuration and need to determine the total raw frames processed:

def actual_frames_from_window(window_size: int,
                             num_scale_frames: int,
                             keyframe_interval: int = 1) -> int:
    """
    Return the total number of raw video frames that will be processed
    inside a single window.

    Parameters
    ----------
    window_size : int
        Number of keyframes (including scale frames) in the window.
    num_scale_frames : int
        Number of scale frames that are processed together.
    keyframe_interval : int, optional
        Interval between retained keyframes in Phase‑2. Default = 1.

    Returns
    -------
    int
        Total number of actual frames covered by the window.
    """
    # Guard against invalid configs

    if num_scale_frames > window_size:
        raise ValueError("num_scale_frames cannot exceed window_size")
    if keyframe_interval < 1:
        raise ValueError("keyframe_interval must be >= 1")

    return num_scale_frames + (window_size - num_scale_frames) * keyframe_interval

Calculating Minimum Window Size for Target Coverage

Use this function when you have a desired temporal coverage and need to compute the minimal window_size:

def min_window_for_coverage(target_frames: int,
                           num_scale_frames: int,
                           keyframe_interval: int = 1) -> int:
    """
    Given a desired actual frame coverage, compute the minimal
    `window_size` (keyframes) needed.

    Parameters
    ----------
    target_frames : int
        Desired number of raw frames per window.
    num_scale_frames : int
        Fixed number of scale frames.
    keyframe_interval : int, optional
        Interval between retained keyframes. Default = 1.

    Returns
    -------
    int
        Minimal window_size (>= num_scale_frames) that satisfies the target.
    """
    if target_frames <= num_scale_frames:
        # Coverage is limited by the scale phase alone.

        return num_scale_frames

    # Solve for window_size in:

    # target = scale + (window - scale) * keyframe_interval

    # => window = scale + ceil((target - scale) / keyframe_interval)

    extra = target_frames - num_scale_frames
    window = num_scale_frames + (extra + keyframe_interval - 1) // keyframe_interval
    return window

Verification Examples

if __name__ == "__main__":
    # 1. Default case – every frame is a keyframe

    assert actual_frames_from_window(window_size=16,
                                    num_scale_frames=4,
                                    keyframe_interval=1) == 16

    # 2. Every 3rd frame kept as keyframe

    assert actual_frames_from_window(window_size=16,
                                    num_scale_frames=4,
                                    keyframe_interval=3) == 4 + (16 - 4) * 3  # = 40

    # 3. Compute minimal window for a target of 50 frames,

    #    with 4 scale frames and a keyframe interval of 3

    assert min_window_for_coverage(target_frames=50,
                                   num_scale_frames=4,
                                   keyframe_interval=3) == 4 + (50 - 4 + 2) // 3  # = 19

Source Code References

The frame coverage logic is implemented in the following key files:

• lingbot_map/models/gct_stream_window_v2.py: Contains the windowed inference implementation and the documented formula at lines 1043-1048.
• scripts/benchmark_gct_memory.py: Demonstrates how to pass --sliding-window and --keyframe-interval arguments that affect the coverage calculation.
• README.md: Provides high-level usage examples for windowed inference.

Summary

• LingBot-Map counts keyframes, not raw frames: The window_size parameter in GCTStream.inference_windowed specifies the number of keyframes (frames with retained KV cache), not the actual video frames processed.
• Apply the coverage formula: Calculate actual frames using actual_frames = scale_frames + (window_size - scale_frames) * keyframe_interval.
• Account for scale frames: The first num_scale_frames are always retained, while Phase-2 frames depend on the keyframe_interval.
• Derive window size from coverage: When you need specific temporal coverage, use the rearranged formula to solve for the minimal window_size that satisfies your requirements.

Frequently Asked Questions

What is the difference between window_size and actual frame coverage in LingBot-Map?

The window_size parameter counts keyframes—frames whose KV cache is retained after processing—whereas actual frame coverage includes all raw video frames processed within the window. When keyframe_interval > 1, the window processes additional frames between keyframes that are not retained in the KV cache, resulting in actual coverage that exceeds the window size.

How do I calculate the minimum window_size needed for a specific temporal coverage?

Use the inverse of the coverage formula: window_size = num_scale_frames + ceil((target_frames - num_scale_frames) / keyframe_interval). This ensures you allocate enough keyframes to cover your desired number of actual frames, rounding up to the nearest whole keyframe since windows must contain complete keyframe entries.

What happens when keyframe_interval is set to 1?

When keyframe_interval equals 1, every Phase-2 frame becomes a keyframe, meaning no frames are discarded from the KV cache. In this configuration, the formula simplifies to actual_frames = window_size, and the window size directly equals the number of actual frames processed.

Where is the frame coverage calculation documented in the source code?

The exact relationship is documented in lingbot_map/models/gct_stream_window_v2.py at lines 1043-1048, where the development team explicitly notes that window_size counts keyframes and provides the formula for calculating actual frame coverage when keyframe_interval > 1.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}