Streaming vs Windowed Inference in LingBot-Map: Key Differences Explained

Streaming inference maintains a single persistent KV cache for continuous frame-by-frame processing, while windowed inference partitions video into independent chunks with fresh caches and applies temporal alignment stitching to merge results.

LingBot-Map (available in the Robbyant/lingbot-map repository) provides two distinct inference regimes through its GCTStream model architecture. Both modes leverage the same underlying KV-cache-enabled aggregator and causal camera head, but they differ fundamentally in sequence handling, memory management, and temporal alignment strategies.

How Streaming Inference Processes Video Sequences

In streaming inference, the model processes the entire video sequence (or the portion that fits in GPU memory) within a single forward pass using one persistent KV cache.

The implementation in inference_streaming (lines 495–564 of lingbot_map/models/gct_stream_window.py) first processes scale frames to establish the spatial reference, then iterates over remaining frames one-by-one. The KV cache grows as keyframes are stored, with non-keyframes optionally skipping cache append via the _set_skip_append mechanism to save memory. The cache can grow up to the configured kv_cache_sliding_window size, after which older entries are evicted.

Keyframe selection in streaming mode supports two strategies:

• Fixed-interval: Store KV entries every keyframe_interval frames
• Flow-based: Compute optical flow against the last keyframe using flow_threshold, deferring eviction and potentially rolling back the cache (_rollback_last_frame) if motion is insufficient

How Windowed Inference Manages Memory and Alignment

Windowed inference addresses long-video memory constraints by splitting the sequence into independent windows, each receiving a fresh KV cache via clean_kv_cache() calls before processing.

The inference_windowed method (lines 590–660 of lingbot_map/models/gct_stream_window.py) processes each window as a mini-streaming session, including its own scale frames and streaming frames. Critical to this approach is the overlap mechanism: each window shares overlap_keyframes with the next window, which become the subsequent window's scale frames. This overlap preserves bidirectional attention continuity across chunk boundaries.

After all windows complete, the model executes temporal alignment stitching through _align_and_stitch_windows (lines 668–756). This process:

1. Estimates similarity transforms (scale, rotation, translation) from overlapping keyframes using _pairwise_alignment
2. Warps predictions into the coordinate frame of the previous window via _warp_predictions
3. Merges results seamlessly using _stitch_windows

Key Differences Between Streaming and Windowed Modes

Aspect	Streaming Inference	Windowed Inference
KV Cache Scope	Single cache persisting across all frames	Fresh cache per window, cleared via clean_kv_cache()
Memory Footprint	Scales with kv_cache_sliding_window and sequence length	Bounded by window_size, constant regardless of video length
Temporal Alignment	None required; native model coordinates	Post-processing alignment using _align_and_stitch_windows
Keyframe Handling	Fixed or flow-based selection within continuous stream	Fixed/flow-based plus overlap frames serving as next window's scale frames
Typical Use Case	Real-time/online processing with frame-by-frame arrival	Offline batch processing of long videos with memory constraints

Memory and Performance Implications

Streaming mode excels in real-time applications where frames arrive continuously and low latency is critical. However, memory usage grows with the number of stored keyframes unless you increase keyframe_interval, which may reduce temporal resolution.

Windowed mode guarantees predictable GPU memory usage even for hour-long videos, as memory is bounded by window_size rather than total frame count. The trade-off is computational overhead from the alignment stitching process and the need to process overlapping frames twice (once per adjacent window).

Practical Implementation: Running Both Inference Modes

Both modes share the same GCTStream class from lingbot_map/models/gct_stream_window.py and accept similar parameters for scale frames and keyframe selection.

import torch
from lingbot_map.models.gct_stream_window import GCTStream

# Initialize model

model = GCTStream(pretrained_path="path/to/checkpoint.pt")
model.eval()

# Sample video tensor: 30 frames, 3×224×224 RGB

images = torch.rand(30, 3, 224, 224)  # Shape [S, C, H, W]

# Streaming inference: single KV cache, continuous processing

stream_output = model.inference_streaming(
    images,
    num_scale_frames=4,
    keyframe_interval=2,
    output_device=torch.device("cpu")
)
print("Streaming output shape:", stream_output["pose_enc"].shape)  # [1, 30, 9]

# Windowed inference: chunked processing with alignment

window_output = model.inference_windowed(
    images,
    window_size=12,
    overlap_keyframes=2,
    keyframe_interval=2,
    flow_threshold=0.0,
    output_device=torch.device("cpu")
)
print("Windowed output shape:", window_output["pose_enc"].shape)  # [1, 30, 9]

The streaming call exercises inference_streaming with its single persistent cache, while the windowed call repeatedly invokes clean_kv_cache() and finishes with _align_and_stitch_windows to fuse predictions.

Summary

• Streaming inference uses one persistent KV cache for the entire sequence, ideal for real-time processing but memory scales with video length.
• Windowed inference splits video into chunks with independent caches, keeping memory bounded by window_size rather than sequence length.
• Windowed mode requires temporal alignment stitching (_align_and_stitch_windows) to merge overlapping window predictions into a coherent trajectory.
• Both modes rely on the shared AggregatorStream and CameraCausalHead architecture from lingbot_map/aggregator/stream.py and lingbot_map/heads/camera_head.py.
• Keyframe selection (fixed-interval or flow-based) works in both modes, but windowed mode adds overlap frames to maintain temporal continuity across chunks.

Frequently Asked Questions

When should I use streaming inference over windowed inference?

Use streaming inference when processing live video streams or when you need continuous, low-latency predictions updated frame-by-frame. According to the LingBot-Map source code, this mode is optimal for online applications where the entire video is not available upfront and you want a single, continuously updated trajectory estimate without post-processing alignment.

How does windowed inference maintain temporal consistency across chunks?

Windowed inference uses overlapping keyframes (overlap_keyframes parameter) where the final frames of one window become the scale frames of the next. After processing, the _align_and_stitch_windows method computes similarity transforms from these overlaps using _pairwise_alignment, then warps predictions into a common coordinate frame via _warp_predictions before final stitching.

What is the maximum video length supported in streaming mode?

Streaming mode has no fixed frame limit, but maximum sequence length is constrained by available GPU memory and the kv_cache_sliding_window configuration. For very long videos (hundreds or thousands of frames), you must either increase keyframe_interval to skip more frames (reducing temporal resolution) or switch to windowed mode to maintain constant memory usage.

Can I adjust the KV cache size in windowed inference?

In windowed inference, the effective KV cache size is determined by window_size (the number of keyframes per window) rather than a separate cache limit. Each window starts with a clean cache via clean_kv_cache(), ensuring memory usage stays predictable regardless of how you configure the sliding window parameter used in streaming mode.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}