Windowed Inference in LingBot-Map: Complete Parameter Guide for Long Video Processing

The inference_windowed method in GCTStream accepts ten parameters—including window_size, overlap_keyframes, and flow_threshold—that control memory-bounded processing of long video sequences by managing keyframe caching and cross-window alignment.

LingBot-Map (available at Robbyant/lingbot-map) implements a windowed inference mode that enables processing arbitrarily long video sequences while keeping GPU memory usage constant. This functionality centers on the inference_windowed method defined in lingbot_map/models/gct_stream_window.py (lines 590–714), which slices input streams into overlapping windows and manages keyframe retention through a sophisticated KV-cache eviction strategy.

Core Parameters for Windowed Inference

The inference_windowed method signature accepts specific arguments that control how video frames are batched, cached, and stitched together across temporal boundaries.

Input Configuration

• images (torch.Tensor): The input image batch with shape [S, 3, H, W] for single-batch inference or [B, S, 3, H, W] for batched processing. Pixel values must be normalized to the range [0, 1].

Window Structure

• window_size (int, default: 16): Specifies the number of keyframes (frames whose KV-cache is retained) per window, including the scale frames. Increasing this value expands the temporal context available to the model but directly increases memory consumption.

• num_scale_frames (int, default: None): Determines how many initial frames in each window are processed as scale frames using bidirectional attention. These provide a reliable depth-scale reference for the remainder of the window. When None, the value falls back to self.num_frame_for_scale.

• scale_mode (str, default: 'median'): Defines the strategy for estimating the inter-window scale factor. Currently, only 'median' is implemented, which calculates the median depth-ratio across paired keyframes to align adjacent windows.

Cross-Window Overlap

• overlap_size (int, default: None): Sets the length of overlapping actual frames between consecutive windows. If unspecified, it defaults to the number of scale frames (num_scale_frames). This overlap ensures the next window receives its scale-phase frames, maintaining prediction quality at boundaries.

• overlap_keyframes (int, default: None): Specifies the overlap expressed in keyframes rather than raw frames. This parameter takes precedence over overlap_size when keyframe_interval > 1. Internally, the method converts this to actual frames as max(num_scale_frames, overlap_keyframes × keyframe_interval).

Keyframe Management Strategy

• keyframe_interval (int, default: 1): Controls the density of cached keyframes. A value of 1 means every frame becomes a keyframe (the original streaming behavior), while larger values reduce cache growth proportionally by retaining only every N-th frame.

• flow_threshold (float, default: 0.0): Enables flow-based dynamic keyframe detection when set greater than zero. The method computes the mean optical-flow magnitude (in pixels) between the current frame and the last keyframe; frames exceeding this threshold become keyframes. When active, this mode overrides the fixed keyframe_interval.

• max_non_keyframe_gap (int, default: 30): In flow-based mode, sets the maximum number of consecutive non-keyframes allowed before a keyframe is forced to prevent temporal drift.

Memory Management

• output_device (torch.device, default: None): Specifies where final predictions are stored. When set to torch.device('cpu'), the method offloads per-window results from GPU memory, enabling processing of very long sequences on limited hardware. If None, predictions remain on the model's current device.

Internal Processing Pipeline

Understanding how these parameters interact helps optimize inference for specific hardware constraints.

The method executes four distinct phases:

1. Window Creation: The algorithm first calculates the effective overlap (eff_overlap) based on overlap_keyframes (preferred) or overlap_size, then partitions the sequence into overlapping chunks.

2. Scale Phase: For each window, the first num_scale_frames frames are processed together with bidirectional attention. Their KV cache is preserved to establish the depth scale reference.

3. Streaming Phase: Remaining frames process sequentially. With keyframe_interval > 1, only every N-th frame enters the cache. When flow_threshold > 0, the method temporarily disables cache eviction via _set_defer_eviction, computes flow magnitude via _compute_flow_magnitude, and conditionally rolls back non-keyframes using _rollback_last_frame.

4. Cross-Window Alignment: After each window completes, _align_and_stitch_windows aligns predictions using the overlapping region via _depth_ratio_scale and pose-based similarity transforms, then stitches them via _stitch_windows.

Implementation Examples

Basic Windowed Inference

import torch
from lingbot_map.models.gct_stream_window import GCTStream

# Initialize model (paths omitted for brevity)

model = GCTStream(
    img_size=518,
    patch_size=14,
    embed_dim=1024,
    pretrained_path="path/to/dinov2_weights.pt",
    enable_camera=True,
    enable_depth=True,
    enable_3d_rope=False,
)
model.eval()
model.cuda()

# Prepare dummy video: 120 frames of 518×518

B, S, C, H, W = 1, 120, 3, 518, 518
dummy_frames = torch.rand(B, S, C, H, W).cuda()  # Values in [0, 1]

# Run windowed inference

output = model.inference_windowed(
    images=dummy_frames,
    window_size=20,                # 20 keyframes per window

    overlap_keyframes=5,           # 5-keyframe overlap

    num_scale_frames=2,            # First 2 frames use bidirectional attention

    keyframe_interval=4,           # Cache every 4th frame

    flow_threshold=0.0,           # Disable flow-based mode

    output_device=torch.device('cpu'),  # Offload to CPU

)

# Extract results

pose_enc = output["pose_enc"]          # [B, S, 9]

depth = output.get("depth")            # [B, S, H, W, 1]

world_pts = output.get("world_points") # [B, S, H, W, 3]

Flow-Based Dynamic Keyframe Detection

output = model.inference_windowed(
    images=dummy_frames,
    window_size=16,
    overlap_size=4,              # 4-frame overlap

    flow_threshold=1.5,          # Activate flow-based mode (pixels)

    max_non_keyframe_gap=10,     # Force keyframe every 10 frames max

)

In this configuration, the model automatically selects keyframes based on optical flow magnitude while respecting the max_non_keyframe_gap constraint to prevent drift.

Summary

• inference_windowed in lingbot_map/models/gct_stream_window.py enables fixed-memory processing of long videos through sliding windows.
• Window boundaries are controlled by window_size, overlap_size, and overlap_keyframes, with the latter preferred when subsampling keyframes.
• Scale frames (num_scale_frames) establish depth references using bidirectional attention within each window.
• Keyframe density is managed either statically via keyframe_interval or dynamically via flow_threshold and max_non_keyframe_gap.
• Memory optimization is achieved through output_device offloading, which moves results to CPU while keeping model weights on GPU.

Frequently Asked Questions

What is the difference between overlap_size and overlap_keyframes?

Both parameters define the overlap between consecutive windows, but overlap_keyframes operates in the keyframe domain while overlap_size operates in raw frames. When keyframe_interval > 1, use overlap_keyframes to ensure the overlap length respects the subsampled keyframe spacing. The method converts overlap_keyframes to actual frames using max(num_scale_frames, overlap_keyframes × keyframe_interval).

How does flow_threshold enable dynamic keyframe detection?

When flow_threshold is greater than zero, the model computes the mean optical-flow magnitude between the current frame and the lastkeyframe using _compute_flow_magnitude. If the displacement exceeds the threshold (in pixels), the frame becomes a keyframe regardless of keyframe_interval. This ensures that rapid motion or scene changes trigger cache retention, while static sequences maintain minimal memory usage.

What happens if num_scale_frames exceeds window_size?

The num_scale_frames parameter must be less than or equal to window_size minus the required overlap. If configured improperly, the method may raise an error or fail to create valid windows because scale frames consume positions within the window budget. The overlap calculation ensures at least num_scale_frames are available for the scale phase while maintaining the specified overlap between windows.

When should I use CPU offloading via output_device?

Set output_device=torch.device('cpu') when processing very long sequences (thousands of frames) that would otherwise exhaust GPU memory storing intermediate pose, depth, and point-cloud tensors. This offloads only the final prediction tensors to system RAM, allowing the model to remain on GPU for accelerated inference while preventing out-of-memory errors during aggregation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}