Understanding the Relationship Between window_size, overlap_keyframes, and Actual Frame Coverage in Lingbot-Map

The relationship between window_size, overlap_keyframes, and actual frame coverage in Lingbot-Map is governed by keyframe_interval multiplication, where window_size counts keyframes but actual coverage depends on scale frames plus interval-expanded differences, while overlap_keyframes converts to actual frames via the same interval to ensure seamless window stitching.

In the Lingbot-Map repository, windowed inference manages memory-efficient processing of long video sequences by sliding a context window across frames. The parameters window_size and overlap_keyframes appear to count frames directly, but they actually operate on keyframes while the actual frame coverage depends on the keyframe_interval and scale frame configuration. Understanding this translation is critical for tuning inference on long-form video without gaps or redundant processing.

How window_size Translates to Actual Frames

The window_size parameter does not represent actual video frames—it counts keyframes that remain in the KV cache (including mandatory scale frames).

According to the docstring in lingbot_map/models/gct_stream_window_v2.py (lines 43-47), the total number of actual frames covered by one window follows this calculation:

actual_frames = scale_frames + (window_size - scale_frames) × keyframe_interval

Where:

• scale_frames (ws in the source) are the base frames always kept in cache
• (window_size - scale_frames) represents the additional keyframe slots
• keyframe_interval (kf in the code) expands each keyframe slot to its representative frame count

For example, with window_size=16, scale_frames=4, and keyframe_interval=2, the window actually covers 4 + (12 × 2) = 28 actual frames, not 16.

Resolving Overlap Parameters

Lingbot-Map accepts overlap specification through two mutually exclusive parameters resolved in a priority order (lines 93-106 of gct_stream_window_v2.py):

1. overlap_keyframes – Overlap expressed in keyframes (preferred modern interface)
2. overlap_size – Legacy parameter specifying overlap in actual frames (used only when overlap_keyframes is absent)

When overlap_keyframes is provided, the code converts it to actual frames at lines 56-60:

# From lines 56-60: converting keyframe overlap to actual frame overlap

overlap_actual = overlap_keyframes × keyframe_interval

This conversion ensures that the overlap region contains at least overlap_keyframes × keyframe_interval actual frames, maintaining temporal continuity regardless of keyframe sampling density.

Effective Overlap Calculation and Constraints

The final overlap value—stored as eff_overlap—undergoes resolution logic between lines 93-106:

Priority resolution:

• If overlap_keyframes exists: eff_overlap = overlap_keyframes × keyframe_interval
• Else if overlap_size exists: eff_overlap = overlap_size
• Else default: eff_overlap = num_scale_frames

Safety constraints:

• Minimum bound: The effective overlap is forced to at least num_scale_frames (line 98), ensuring the next window retains sufficient scale frames for reliable alignment regardless of user input.
• Maximum bound: The value is clamped to S-1 (total sequence length minus one) at line 106 to prevent cursor overshoot.

The cursor advancement follows:

cursor = max(cursor - eff_overlap, window_start + window_scale)

This means each new window starts eff_overlap frames back from the previous window's end, creating the stitching region.

Complete Coverage Formula

The actual coverage per inference step combines window size and overlap:

Total unique coverage per window = scale_frames + (window_size - scale_frames) × keyframe_interval
Effective overlap in actual frames = max(overlap_keyframes × keyframe_interval, scale_frames)

When overlap_keyframes is set, the shared actual frames equal overlap_keyframes × keyframe_interval (subject to the minimum scale frame constraint). When using legacy overlap_size, the value applies directly without interval multiplication.

Practical Implementation Examples

The following examples demonstrate how these parameters translate in demo.py (lines 396-401) and the core inference logic:

# Example 1: Default keyframe_interval=1

# 16 keyframes per window, 8 overlapping actual frames

pred = model.inference_windowed(
    images,               # Tensor shape [S, 3, H, W]

    window_size=16,
    overlap_keyframes=8,  # Maps directly to 8 actual frames when interval=1

)

# Example 2: Keyframe interval > 1 (e.g., interval=2)

# 16 keyframes per window with 4 keyframes overlap

# Actual overlap becomes 4 × 2 = 8 frames

pred = model.inference_windowed(
    images,
    window_size=16,
    overlap_keyframes=4,
    keyframe_interval=2,
)

# Example 3: Legacy overlap_size usage

# Explicit actual frame overlap without keyframe conversion

pred = model.inference_windowed(
    images,
    window_size=16,
    overlap_size=12,  # Exactly 12 actual frames overlap

)

In each case, the model internally resolves eff_overlap using the priority logic described above and stitches windows accordingly.

Key Implementation Files

File	Role
lingbot_map/models/gct_stream_window_v2.py	Core windowed inference implementation containing the resolution logic for window_size, overlap_keyframes, and overlap_size (lines 43-47, 53-60, 93-106).
lingbot_map/models/gct_stream_window.py	Previous implementation version providing historical context for the windowing logic.
demo.py	Command-line interface exposing --overlap_size and --overlap_keyframes arguments (lines 396-401).

Summary

• window_size counts keyframes, not actual frames; actual coverage equals scale_frames + (window_size - scale_frames) × keyframe_interval.
• overlap_keyframes is multiplied by keyframe_interval to determine actual frame overlap, while overlap_size specifies actual frames directly.
• Effective overlap is determined by priority: overlap_keyframes > overlap_size > default num_scale_frames.
• Safety constraints enforce minimum overlap of scale_frames and maximum of S-1 total frames.
• These relationships ensure that windowed inference in Lingbot-Map maintains temporal continuity while managing KV cache memory efficiently.

Frequently Asked Questions

What is the difference between overlap_keyframes and overlap_size?

overlap_keyframes measures overlap in keyframe units and is converted to actual frames by multiplying with keyframe_interval, while overlap_size measures overlap directly in actual frames without conversion. According to lines 53-60 of gct_stream_window_v2.py, overlap_keyframes takes precedence when both are provided, and overlap_size serves as a legacy fallback.

How does keyframe_interval affect actual frame coverage?

The keyframe_interval acts as a multiplier between keyframe counts and actual frame counts. As documented at lines 43-47, each keyframe beyond the scale frames represents keyframe_interval actual frames. When set to 1, keyframes and actual frames correspond 1:1; when greater than 1, a single keyframe stands for multiple actual frames, expanding both window coverage and overlap regions proportionally.

Why is effective overlap clamped to S-1?

The effective overlap (eff_overlap) is clamped to S-1 (total sequence length minus one) at line 106 to prevent the sliding cursor from attempting to start a new window beyond the sequence boundary or creating negative indexing. This ensures the window stepping logic remains valid regardless of parameter configuration or sequence length.

What happens if overlap_keyframes is smaller than scale frames?

If the calculated overlap from overlap_keyframes would result in fewer actual frames than num_scale_frames, the code at line 98 forces eff_overlap to equal num_scale_frames. This guarantees that every new window retains the minimum required scale frames necessary for reliable spatial alignment, overriding smaller user-specified values to maintain inference quality.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}