How Windowed Inference Handles Long Sequences in LingBot-Map

LingBot-Map processes arbitrarily long video sequences by dividing them into overlapping windows with bounded KV caches, running a bidirectional scale-phase followed by causal streaming, and stitching results using geometric alignment of overlapping keyframes.

LingBot-Map implements a sliding-window inference pipeline that keeps GPU memory constant regardless of input length. This approach, implemented in the GCTStream class within the Robbyant/lingbot-map repository, enables high-quality pose and depth estimation on videos containing millions of frames without running out of memory.

Dividing Sequences into Overlapping Windows

The windowed inference algorithm begins in the inference_windowed method (lines ~1200-1260 of lingbot_map/models/gct_stream_window_v2.py) by partitioning the video stream into manageable chunks.

Each window contains a fixed number of keyframes (stored in the KV cache) plus scale frames that are processed bidirectionally. The window_size parameter defaults to 16 keyframes, but the actual number of frames per window follows this formula:

actual_frames = scale_frames + (window_size - scale_frames) × keyframe_interval

Consecutive windows overlap by at least the number of scale frames (overlap_size defaults to num_scale_frames). You can also specify overlap in keyframes using overlap_keyframes, which the method internally converts to actual-frame overlap using the keyframe interval.

KV Cache Management and Memory Bounding

Before processing each window, the model calls clean_kv_cache() to clear both the aggregator's and camera head's key-value caches. This guarantees that memory usage never exceeds the size required for a single window, regardless of total video length.

The AggregatorStream class in lingbot_map/aggregator/stream.py implements the underlying cache logic (supporting FlashInfer or SDPA backends) and provides the eviction, rollback, and defer-eviction mechanisms used during streaming.

Two-Phase Processing Within Windows

Scale-Phase (Phase 1)

The first num_frame_per_block frames (equal to scale_frames) run together as a batch. Because these frames share a scale token, they attend bidirectionally rather than causally. This yields a robust estimate of the scene scale that improves downstream alignment between windows.

Streaming Phase (Phase 2)

The remaining frames in the window are processed one-by-one in causal mode:

• Keyframe mode (keyframe_interval > 1): Only every Nth frame becomes a keyframe and retains its KV cache entry. Non-keyframes are processed with skip_append=True so their KV is discarded immediately after the forward pass (see _set_skip_append).

• Flow-based mode (flow_threshold > 0): The model computes optical flow magnitude between the current frame and the latest keyframe using _compute_flow_magnitude. If the flow exceeds the threshold, the frame becomes a keyframe; otherwise, the model calls _rollback_last_frame and processes it with deferred eviction via _set_defer_eviction.

The causal camera head in lingbot_map/heads/camera_head.py respects these KV-cache flags during per-frame pose refinement.

Global Alignment and Result Stitching

After all windows are evaluated, the pipeline reconciles them into a globally consistent reconstruction:

1. Pairwise alignment: _pairwise_alignment estimates similarity transforms (scale s, rotation R, translation t) using the overlapping keyframes between adjacent windows.

2. Warping: Each window is warped into the coordinate frame of the first window using _warp_predictions.

3. Stitching: Results are concatenated via _stitch_windows, which de-duplicates overlapping frames to produce a seamless output.

The final merged prediction dictionary contains the full-length pose encodings, depth maps, world-point clouds, and the original images with shapes preserved across the entire sequence.

Device Handling and Long-Sequence Optimization

For extremely long sequences, the inference_windowed method accepts an output_device parameter (e.g., torch.device('cpu')). This offloads per-frame outputs from GPU to CPU memory during processing, preventing out-of-memory errors while keeping the model weights on the accelerator for fast inference.

Practical Implementation Example

import torch
from lingbot_map.models.gct_stream_window_v2 import GCTStream

# Build the model (use a pretrained DINOv2 checkpoint if you have one)

model = GCTStream(
    img_size=518,
    patch_embed='dinov2_vitl14_reg',
    sliding_window_size=-1,          # full causal by default

    num_frame_for_scale=8,           # 8 scale frames per window

    enable_camera=True,
)

# Dummy video: 500 frames, 3×518×518 RGB, batch size 1

video = torch.rand(1, 500, 3, 518, 518)

# Run windowed inference:

preds = model.inference_windowed(
    images=video,
    window_size=16,          # 16 keyframes per window

    overlap_keyframes=4,     # 4‑keyframe overlap → 8 actual‑frame overlap

    keyframe_interval=2,     # store KV every other frame

    flow_threshold=0.0,      # disable flow‑based mode

    output_device=torch.device('cpu')  # keep results on CPU

)

# `preds` now contains full‑length pose_enc, depth, world_points, etc.

print(preds["pose_enc"].shape)   # → torch.Size([1, 500, 9])

Summary

• Windowed inference divides long sequences into overlapping chunks with configurable keyframe and scale-frame counts.
• Bounded memory is achieved by clearing the KV cache (clean_kv_cache()) before each window processes.
• Bidirectional scale estimation in Phase 1 establishes robust scene scale, while causal streaming in Phase 2 handles the remaining frames with selective KV retention.
• Geometric alignment via _pairwise_alignment and _warp_predictions stitches windows into a globally consistent coordinate frame.
• CPU offloading via output_device allows processing of millions of frames without GPU memory exhaustion.

Frequently Asked Questions

What is the maximum video length LingBot-Map can process?

There is no fixed maximum length. Because windowed inference keeps GPU memory bounded to the size of a single window (controlled by window_size and num_frame_for_scale), the system can theoretically process millions of frames, limited only by available CPU RAM for storing results when output_device is set to CPU.

How does the keyframe interval affect reconstruction quality?

A smaller keyframe_interval stores more frames in the KV cache, improving temporal coherence and accuracy but increasing memory usage per window. A larger interval reduces memory but may miss rapid motion. The flow-based mode (flow_threshold > 0) provides an adaptive alternative that inserts keyframes dynamically based on optical flow magnitude computed in _compute_flow_magnitude.

What is the difference between scale frames and keyframes?

Scale frames (handled by num_frame_for_scale) are the first frames in each window that run bidirectionally to establish scene scale. Keyframes (controlled by window_size and keyframe_interval) are retained in the KV cache throughout the window to maintain temporal context; non-keyframes are processed with skip_append=True and discarded immediately to save memory.

Can windowed inference run on CPU-only machines?

Yes, by setting output_device=torch.device('cpu') and ensuring the model itself is on CPU, the pipeline can run without a GPU. However, inference will be significantly slower as the GCTStream architecture relies on accelerated attention backends (FlashInfer or SDPA) for efficient processing of the scale and streaming phases.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}