How to Implement Windowed Inference Mode for Video Sequences Longer Than 3000 Frames

To process videos longer than 3000 frames without GPU out-of-memory errors, call GCTStream.inference_windowed() in lingbot_map/models/gct_stream_window_v2.py, which splits the input into overlapping windows, resets the KV cache per window, dynamically selects keyframes, and stitches results with a similarity transform estimated from shared overlap frames.

Ling-Bot-Map, available at Robbyant/lingbot-map, is an open-source spatial-reasoning system that normally processes video with causal attention and a KV cache. While effective for short clips, this cache grows linearly with frame count and exhausts GPU memory on long sequences. Implementing windowed inference mode for video sequences bounds memory usage to a single window, enabling inference on 10,000-frame videos or longer.

Why Standard Streaming Inference Fails Beyond 3000 Frames

Standard causal streaming retains every keyframe in the KV cache so that new frames can attend to all prior context. On a 5,000- or 10,000-frame video, this unbounded cache growth triggers an out-of-memory error regardless of how small the batch size is. The repository solves this by providing a windowed alternative that resets the cache at regular intervals while preserving geometric continuity across segments.

Core Architecture of Windowed Inference

The windowed strategy implemented in lingbot_map/models/gct_stream_window_v2.py rests on four pillars that keep memory constant regardless of total video length.

Overlapping windows with a fresh KV cache — The video is divided into windows of window_size keyframes. Before each window begins, self.clean_kv_cache() wipes the previous context, ensuring the cache never exceeds kv_cache_sliding_window entries. An overlap region between consecutive windows provides common frames for later geometric alignment.

Bidirectional scale frames — Every window starts with a small set of scale frames that are processed together. These frames receive a full forward pass with num_frame_per_block=window_scale to create a shared spatial reference. The rest of the window then proceeds with standard causal streaming.

Keyframe selection — Only selected frames populate the KV cache. The model supports two modes:

• Fixed-interval mode: A frame is stored every keyframe_interval steps. Non-keyframes call self._set_skip_append(True) to bypass cache insertion.
• Flow-based mode: When flow_threshold > 0, the model defers eviction with self._set_defer_eviction(True), computes optical flow magnitude via _compute_flow_magnitude(), and promotes high-motion frames to keyframes. The max_non_keyframe_gap parameter forces a keyframe after long static stretches.

Cross-window alignment and stitching — After all windows finish, the pipeline calls _align_and_stitch_windows(). Internally, _pairwise_alignment() estimates a similarity transform (scale, rotation, translation) from overlapping keyframes; _warp_predictions() applies it; and _stitch_windows() concatenates the sequences while discarding duplicated overlap frames.

Inside the inference_windowed() Pipeline

The following walkthrough traces the exact operations performed by GCTStream.inference_windowed() in lingbot_map/models/gct_stream_window_v2.py.

Input Normalization and Overlap Setup

At lines 1024–1028, the method ensures a batch dimension exists:

if len(images.shape) == 4:
    images = images.unsqueeze(0)

Between lines 1093–1101, the overlap is resolved. If overlap_keyframes is provided, it is converted to a raw frame count using keyframe_interval. If overlap_size is provided, it is used directly. Otherwise, the effective overlap defaults to the window size.

Phase 1 – Processing Scale Frames

Lines 1159–1170 process the scale frames to obtain bidirectional context before causal streaming starts:

scale_out = self.forward(
    ...,
    num_frame_per_block=window_scale,
    causal_inference=True
)

Phase 2 – Streaming Frames and Keyframe Detection

For the remaining frames in the window, the model switches to causal one-by-one processing.

If flow-based keyframe detection is enabled (flow_threshold > 0), the logic at lines 1196–1229 performs the following:

1. Defer cache eviction with self._set_defer_eviction(True).
2. Run the forward pass.
3. Compute optical flow magnitude using _compute_flow_magnitude().
4. Compare against flow_threshold to decide keyframe status.

If fixed-interval mode is active, lines 1245–1257 toggle self._set_skip_append(True) for non-keyframes so they do not enlarge the KV cache.

Prediction Collection and Memory Cleanup

Per-frame outputs are optionally moved to output_device to prevent GPU accumulation. Lines 1285–1315 accumulate predictions in Python lists via an internal _to_out helper. After the final window, lines 1361–1368 clear the KV cache again and call torch.cuda.empty_cache() if CUDA is available.

Cross-Window Alignment and Stitching

The post-processing block at lines 1443–1505 executes:

• _pairwise_alignment() on overlapping keyframes to estimate the transform.
• _warp_predictions() to geometrically align neighboring windows.
• _stitch_windows() to merge sequences and drop redundant overlap.

The merged dictionary returned at lines 1510–1519 contains pose_enc, depth, and world_points.

Memory Guarantees for Arbitrary-Length Videos

Because each window resets the KV cache, memory consumption is bounded by the cache capacity of a single window rather than the full video length. Setting kv_cache_sliding_window=64 ensures that even a 10,000-frame video consumes the same peak VRAM as a 64-frame clip. For extremely long sequences, set output_device=torch.device('cpu') to stream per-frame tensors to host memory and keep the GPU footprint minimal.

Practical Code Examples

Long Video with Dynamic Keyframes

import torch
from lingbot_map.models.gct_stream_window_v2 import GCTStream

model = GCTStream(
    img_size=518,
    patch_embed='dinov2_vitl14_reg',
    enable_camera=True,
    enable_point=True,
    enable_depth=True,
    sliding_window_size=-1,
    kv_cache_sliding_window=64,
    kv_cache_scale_frames=8,
)
model.eval()
model.to('cuda')

video = torch.randn(5000, 3, 518, 518).to('cuda')

pred = model.inference_windowed(
    images=video,
    window_size=16,
    overlap_keyframes=4,
    keyframe_interval=5,
    flow_threshold=0.8,
    max_non_keyframe_gap=30,
    output_device=torch.device('cpu'),
)

pose_enc = pred["pose_enc"]
depth_maps = pred["depth"]

Fixed-Interval Mode Only

pred = model.inference_windowed(
    images=video,
    window_size=12,
    overlap_size=8,
    keyframe_interval=3,
    flow_threshold=0.0,
    output_device=None,
)

CPU-Only Execution

model.to('cpu')
pred = model.inference_windowed(
    images=video.cpu(),
    window_size=10,
    overlap_keyframes=2,
    keyframe_interval=1,
    output_device=torch.device('cpu'),
)

Key Files in the Windowed Inference Stack

• lingbot_map/models/gct_stream_window_v2.py — Hosts GCTStream.inference_windowed(), window stitching, and flow-based keyframe logic.
• lingbot_map/models/gct_stream.py — Base streaming inference method that the windowed implementation extends.
• lingbot_map/aggregator/stream.py — AggregatorStream, the transformer backbone that manages the KV cache.
• lingbot_map/heads/camera_head.py — CameraCausalHead, which participates in cache skip and rollback operations.
• lingbot_map/utils/pose_enc.py — Pose encoding utilities used by the flow magnitude computation.
• lingbot_map/utils/geometry.py — Provides closed_form_inverse_se3 and helpers for the similarity transform.
• lingbot_map/models/gct_base.py — Base class defining common forward-pass logic and prediction normalization.

Summary

• Split the video into overlapping windows and reset the KV cache between each window to bound GPU memory.
• Use bidirectional scale frames at the start of every window to preserve spatial consistency.
• Choose between fixed-interval and flow-based keyframe detection to control cache population.
• Offload per-frame outputs to CPU via output_device when processing very long sequences.
• Align and stitch windows with _pairwise_alignment(), _warp_predictions(), and _stitch_windows() to produce a single coherent prediction stream.

Frequently Asked Questions

What is the maximum video length supported by windowed inference mode?

There is no hardcoded frame limit. Because memory usage is bounded per window, the method scales to videos with 10,000 frames or more, constrained only by disk space for the final output tensors and available CPU RAM if output_device is set to CPU.

How does the KV cache stay bounded when processing thousands of frames?

GCTStream.inference_windowed() calls self.clean_kv_cache() before each new window. The cache only retains entries for the current window, capped by kv_cache_sliding_window (default 64). This design prevents the linear growth that causes standard streaming inference to fail.

Should I use flow_threshold or keyframe_interval for keyframe selection?

Use flow_threshold > 0 when scene motion is uneven and you want the model to dynamically retain informative frames. Use flow_threshold = 0.0 with a fixed keyframe_interval for deterministic behavior and slightly lower per-frame overhead. Both modes respect max_non_keyframe_gap to avoid losing tracking during long static shots.

Can I run windowed inference on a CPU-only machine?

Yes. Move the model and inputs to CPU, set output_device=torch.device('cpu'), and use a small window_size so that each forward pass fits in system memory. The windowed design keeps the working set small regardless of total video length.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}