How Windowed Inference Works in LingBot-Map for Processing Very Long Sequences

Windowed inference in LingBot-Map processes arbitrarily long video sequences by dividing them into overlapping windows with fresh KV caches per window, enabling constant GPU memory usage while maintaining global consistency through scale-phase initialization and alignment-based stitching.

The Robbyant/lingbot-map repository implements a sophisticated windowed inference pipeline that allows the model to handle video streams containing millions of frames without exhausting GPU memory. This approach keeps memory usage bounded to the size of a single window while producing globally consistent pose and depth predictions. The core implementation resides in lingbot_map/models/gct_stream_window_v2.py, which orchestrates the division, processing, and stitching of video segments.

The Windowed Inference Architecture

The pipeline treats long sequences as a series of manageable chunks rather than processing them monolithically. Each window contains a fixed number of keyframes stored in the KV cache plus additional scale frames processed bidirectionally to establish scene geometry.

Window Construction and Overlap Management

The inference_windowed method (lines approximately 1200–1260 in lingbot_map/models/gct_stream_window_v2.py) constructs windows using the window_size parameter, which defaults to 16 keyframes. The actual frame count per window follows this calculation:

actual_frames = scale_frames + (window_size - scale_frames) × keyframe_interval

Consecutive windows overlap by at least the number of scale frames (overlap_size defaults to num_scale_frames). Alternatively, you can specify overlap_keyframes, which the system converts to actual-frame overlap using the keyframe interval. This overlap ensures continuity between adjacent segments and enables the alignment phase to compute similarity transforms.

Memory-Bounded KV Cache Strategy

Before processing each window, the model calls clean_kv_cache() to clear both the aggregator’s and camera head’s key-value caches. According to the source code in lingbot_map/models/gct_stream_window_v2.py, this guarantees that GPU memory consumption never exceeds the requirements for a single window, regardless of total video length. The AggregatorStream class in lingbot_map/aggregator/stream.py manages this cache using either FlashInfer or SDPA backends, supporting eviction, rollback, and defer-eviction flags.

Phase 1: Scale Initialization

The first num_scale_frames frames (default 8) of each window run together with num_frame_per_block = num_scale_frames. Because these frames share a scale token, they attend bidirectionally rather than causally. This yields a robust estimate of scene scale and improves downstream alignment quality. The scale phase establishes the coordinate framework before the streaming phase begins.

Phase 2: Streaming with Keyframe Selection

After scale initialization, the remaining frames process one-by-one using two mutually exclusive modes:

Fixed Interval Mode (keyframe_interval > 1): The model retains KV cache entries only for every N-th frame (keyframes). Non-keyframes process with skip_append=True (via _set_skip_append), discarding their KV tensors immediately after the forward pass.

Flow-Based Mode (flow_threshold > 0): The system computes optical flow magnitude between the current frame and latest keyframe using _compute_flow_magnitude. If flow exceeds the threshold, the frame becomes a keyframe; otherwise, _rollback_last_frame reverses the operation after processing with deferred eviction (_set_defer_eviction).

The CameraHead in lingbot_map/heads/camera_head.py respects these KV-cache flags during causal pose refinement.

Global Alignment and Stitching

After window processing, the system reconciles predictions across boundaries to ensure global consistency. The _pairwise_alignment method estimates similarity transforms (scale s, rotation R, translation t) using overlapping keyframes between adjacent windows. Each window then undergoes _warp_predictions to transform into the coordinate frame of the first window. Finally, _stitch_windows concatenates results while de-duplicating overlapping frames, producing a merged prediction dictionary containing full-length pose encodings, depth maps, and world-point clouds.

Implementation Example

The following example demonstrates windowed inference on a 500-frame sequence:

import torch
from lingbot_map.models.gct_stream_window_v2 import GCTStream

# Build the model (use a pretrained DINOv2 checkpoint if you have one)

model = GCTStream(
    img_size=518,
    patch_embed='dinov2_vitl14_reg',
    sliding_window_size=-1,          # full causal by default

    num_frame_for_scale=8,           # 8 scale frames per window

    enable_camera=True,
)

# Dummy video: 500 frames, 3×518×518 RGB, batch size 1

video = torch.rand(1, 500, 3, 518, 518)

# Run windowed inference:

preds = model.inference_windowed(
    images=video,
    window_size=16,          # 16 keyframes per window

    overlap_keyframes=4,     # 4-keyframe overlap → 8 actual-frame overlap

    keyframe_interval=2,     # store KV every other frame

    flow_threshold=0.0,      # disable flow-based mode

    output_device=torch.device('cpu')  # keep results on CPU

)

# `preds` now contains full-length pose_enc, depth, world_points, etc.

print(preds["pose_enc"].shape)   # → torch.Size([1, 500, 9])

Key Source Files and Components

The windowed inference system relies on several specialized modules:

• lingbot_map/models/gct_stream_window_v2.py: Main implementation containing inference_windowed, KV-cache management, and alignment logic.
• lingbot_map/models/gct_stream_window.py: Earlier version of the streaming window class.
• lingbot_map/aggregator/stream.py: Implements AggregatorStream with FlashInfer/SDPA backends supporting cache eviction and rollback.
• lingbot_map/heads/camera_head.py: Causal camera head providing per-frame pose refinement respecting KV-cache flags.
• lingbot_map/utils/pose_enc.py: Utilities for converting pose encodings to extrinsic/intrinsic matrices.
• lingbot_map/utils/geometry.py: Contains closed_form_inverse_se3 for inverting camera transforms during flow computation.

Summary

• Windowed inference divides arbitrarily long sequences into overlapping windows with bounded GPU memory usage.
• Each window begins with a scale phase using bidirectional attention to establish robust scene geometry.
• The streaming phase processes remaining frames using either fixed keyframe intervals or optical flow thresholds to manage cache size.
• Fresh KV caches per window via clean_kv_cache() ensure memory consumption remains constant regardless of video length.
• Global alignment using _pairwise_alignment and _warp_predictions stitches windows into a consistent coordinate frame.
• The system supports device offloading via output_device to prevent OOM errors on extremely long sequences.

Frequently Asked Questions

How does windowed inference prevent GPU out-of-memory errors on long videos?

By calling clean_kv_cache() before processing each window, the system resets the KV cache to empty, ensuring GPU memory usage never exceeds the size required for a single window. This bounds memory consumption to O(window_size) rather than O(sequence_length), allowing processing of millions of frames.

What is the difference between scale frames and keyframes in the windowed inference pipeline?

Scale frames are the initial frames in each window processed bidirectionally with shared scale tokens to establish scene geometry. Keyframes are selected frames (based on interval or optical flow) whose KV tensors are retained in the cache for causal attention during the streaming phase.

How does the system maintain global consistency between separate windows?

After processing all windows, the system uses _pairwise_alignment to compute similarity transforms between overlapping regions, then warps predictions into a common coordinate frame using _warp_predictions. The _stitch_windows method merges results while removing duplicate overlapping frames.

Can I use optical flow instead of fixed intervals for keyframe selection?

Yes, by setting flow_threshold > 0 and keyframe_interval=1, the system enters flow-based mode. It computes optical flow magnitude between current and last keyframe frames using _compute_flow_magnitude, promoting frames to keyframes only when motion exceeds the threshold, and rolling back others via _rollback_last_frame.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}