Windowed Inference Parameters in LingBot-Map: Complete API Reference

The inference_windowed method in LingBot-Map accepts ten configurable parameters—including window_size, overlap_keyframes, flow_threshold, and num_scale_frames—that enable memory-efficient processing of arbitrarily long video sequences through sliding window attention and intelligent KV-cache management.

LingBot-Map is an open-source video understanding framework that implements streaming geometry estimation for long-form content. For processing extended video sequences without exhausting GPU memory, the library provides a specialized windowed inference mode via the GCTStream class, specifically implemented in lingbot_map/models/gct_stream_window.py.

Core Parameters for Windowed Inference

The inference_windowed method defined at lines 590–714 in lingbot_map/models/gct_stream_window.py exposes the following parameters:

Input and Output Configuration

• images (torch.Tensor): The input video batch. Accepts shapes [S, 3, H, W] for single batch or [B, S, 3, H, W] for multiple batches. Pixel values must be normalized to the range [0, 1].

• output_device (torch.device, optional): Specifies where final predictions are stored. Defaults to None, which keeps tensors on the model's device. Setting to torch.device('cpu') enables CPU offloading to reduce GPU memory pressure during long sequences.

Window Configuration

• window_size (int, default 16): Defines the number of keyframes (frames whose KV-cache is retained) processed per window, including scale frames. Larger values increase temporal context but raise memory consumption linearly.

• overlap_size (int, optional): Specifies the length of overlapping actual frames between consecutive windows. If omitted, defaults to the number of scale frames (num_scale_frames). This overlap ensures high-quality predictions at window boundaries by supplying the next window with its scale-phase frames.

• overlap_keyframes (int, optional): Alternative overlap specification expressed in keyframes, preferred when keyframe_interval > 1. Takes precedence over overlap_size. Internally converted to actual frames as max(num_scale_frames, overlap_keyframes × keyframe_interval).

• num_scale_frames (int, optional): Defaults to self.num_frame_for_scale. Defines how many initial frames within each window are processed as scale frames using bidirectional attention, providing a reliable depth-scale reference for subsequent frames.

Keyframe Management

• keyframe_interval (int, default 1): Determines the frequency of Phase-2 frames becoming keyframes. A value of 1 retains every frame in the KV-cache (original streaming behavior), while larger values reduce cache growth proportionally.

• flow_threshold (float, default 0.0): Enables dynamic keyframe detection when greater than zero. Calculates the mean optical-flow magnitude in pixels between the current frame and last keyframe; frames exceeding this threshold become keyframes, overriding keyframe_interval.

• max_non_keyframe_gap (int, default 30): In flow-based mode, sets the maximum allowed consecutive non-keyframes before forcing a keyframe to prevent drift.

Scale Estimation

• scale_mode (str, default 'median'): Strategy for estimating inter-window scale factors. Currently implements 'median', which uses the median depth-ratio across paired keyframes for alignment.

How Parameters Interact During Execution

Understanding the internal flow of inference_windowed helps optimize parameter selection for specific video lengths and memory constraints.

Window Creation and Scale Phase

The method first determines the effective overlap (eff_overlap) based on overlap_keyframes (preferred) or overlap_size. For each window, the first num_scale_frames frames are processed together as scale_images with bidirectional attention. Their KV cache is not cleared, enabling cross-scale-frame communication that establishes the depth reference.

Streaming Phase and Dynamic Keyframes

Remaining frames are processed sequentially. When keyframe_interval exceeds 1, only every N-th frame persists in the KV-cache. If flow_threshold is greater than 0.0, the system temporarily disables cache eviction via _set_defer_eviction, computes flow magnitude using _compute_flow_magnitude, and conditionally commits frames based on motion. Non-keyframes trigger _rollback_last_frame to free cache space immediately.

Cross-Window Alignment

After each window completes, _align_and_stitch_windows aligns adjacent predictions using the overlapping region. The _stitch_windows function combines results using depth-ratio scaling (_depth_ratio_scale) and pose-based similarity transforms to maintain global consistency.

Practical Implementation Examples

Basic Windowed Inference with Fixed Intervals

The following example demonstrates processing a 120-frame video with explicit window sizing and keyframe intervals:

import torch
from lingbot_map.models.gct_stream_window import GCTStream

# Initialize model

model = GCTStream(
    img_size=518,
    patch_size=14,
    embed_dim=1024,
    pretrained_path="path/to/dinov2_weights.pt",
    enable_camera=True,
    enable_depth=True,
    enable_3d_rope=False,
)

model.eval()
model.cuda()

# Prepare dummy video: 120 frames of 3×518×518

B, S, C, H, W = 1, 120, 3, 518, 518
dummy_frames = torch.rand(B, S, C, H, W).cuda()  # Values in [0, 1]

# Run windowed inference

output = model.inference_windowed(
    images=dummy_frames,
    window_size=20,                # 20 keyframes per window

    overlap_keyframes=5,           # 5-keyframe overlap

    num_scale_frames=2,            # First 2 frames are scale frames

    keyframe_interval=4,           # Keep KV every 4th frame

    flow_threshold=0.0,           # Disable flow-based mode

    output_device=torch.device('cpu'),  # Offload to CPU

)

# Extract results

pose_enc = output["pose_enc"]          # [B, S, 9]

depth    = output.get("depth")         # [B, S, H, W, 1]

world_pts = output.get("world_points") # [B, S, H, W, 3]

Flow-Based Dynamic Keyframe Detection

For scenes with variable motion, enable optical-flow-based keyframe selection:

output = model.inference_windowed(
    images=dummy_frames,
    window_size=16,
    overlap_size=4,          # 4-frame overlap

    flow_threshold=1.5,     # Activate flow-based mode (pixel units)

    max_non_keyframe_gap=10,
)

In this configuration, the model automatically promotes high-motion frames to keyframes while respecting the overlap constraints, overriding the fixed keyframe_interval.

Summary

• Windowed inference in LingBot-Map processes long videos through overlapping windows to maintain bounded memory usage, controlled by the inference_windowed method in lingbot_map/models/gct_stream_window.py.
• Memory management relies on window_size, keyframe_interval, and flow_threshold to determine which frames retain KV-cache entries, while output_device enables CPU offloading.
• Overlap parameters (overlap_size vs overlap_keyframes) ensure continuity between windows, with num_scale_frames providing bidirectional depth references at window starts.
• Dynamic keyframes triggered by flow_threshold adapt to motion in the scene, with max_non_keyframe_gap preventing drift during static segments.

Frequently Asked Questions

What is the difference between overlap_size and overlap_keyframes?

overlap_size specifies the actual number of frames to overlap between windows, while overlap_keyframes expresses this overlap in terms of keyframe counts. When keyframe_interval > 1, use overlap_keyframes for more intuitive control; it takes precedence over overlap_size and is converted to actual frames via max(num_scale_frames, overlap_keyframes × keyframe_interval).

How does flow_threshold affect memory usage?

Setting flow_threshold greater than 0.0 enables dynamic keyframe detection based on optical flow magnitude. High-motion frames become keyframes and retain KV-cache entries, while static frames are evicted immediately via _rollback_last_frame. This can reduce memory usage in static scenes but may increase it during high-motion sequences compared to fixed keyframe_interval sampling.

What happens if I set keyframe_interval to 1?

A keyframe_interval of 1 retains every frame in the KV-cache, matching the original streaming behavior of the base GCTStream class. This provides maximum temporal context but linearly increases memory consumption with sequence length, negating the memory-bounded benefits of windowed inference for very long videos.

Where is the windowed inference logic implemented?

The core implementation resides in lingbot_map/models/gct_stream_window.py within the GCTStream.inference_windowed method (lines 590–714). Supporting utilities include lingbot_map/aggregator/stream.py for KV-cache backend management, lingbot_map/utils/pose_enc.py for pose encoding utilities, and lingbot_map/utils/geometry.py for geometric transformations used in cross-window alignment.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}