Streaming vs. Windowed Inference Modes in Ling Bot-Map: A Technical Comparison

Streaming inference maintains a single persistent KV cache across the entire video sequence for real-time processing, whereas windowed inference divides sequences into independent chunks with fresh KV caches and stitches predictions using temporal alignment to maintain bounded memory usage.

Ling Bot-Map implements geometric camera tracking through the GCTStream model, which supports two distinct inference regimes. Both modes utilize the same underlying architecture—specifically the KV-cache-enabled AggregatorStream and causal CameraCausalHead—but they differ fundamentally in how they partition video sequences, manage cache memory, and align temporal predictions.

How Sequence Handling Differs

The primary distinction between these modes lies in how video frames are fed through the model's attention mechanism.

Streaming Mode: Single-Pass Processing

In streaming inference, the model processes the entire sequence (or as much as fits in memory) in one continuous forward pass. According to the source code in lingbot_map/models/gct_stream_window.py (lines 495-564), the inference_streaming() method first processes scale frames to establish the coordinate system, then feeds each subsequent frame individually while reusing the existing KV cache.

This approach maintains temporal continuity naturally because the cache persists across all frames, allowing the model to reference any previously stored keyframe.

Windowed Mode: Chunked Processing with Overlap

Windowed inference splits the sequence into discrete chunks, each receiving an independent KV cache. As implemented in lines 590-660 of gct_stream_window.py, the inference_windowed() method calls clean_kv_cache() at the start of each window, ensuring that memory usage never exceeds the size required for a single chunk.

Critically, the windowed approach adds overlap frames between chunks. These overlapping keyframes become the next window's scale frames, preserving the bidirectional attention benefits of scale frames across window boundaries.

KV Cache Management and Memory Characteristics

The KV cache implementation represents the core technical difference between these inference modes.

Streaming inference utilizes a single KV cache that lives for the entire run. Keyframes (determined by keyframe_interval or flow_threshold) are stored in the cache, while non-keyframes can skip cache appending via the _set_skip_append mechanism. The cache grows up to the configured kv_cache_sliding_window size, meaning memory footprint scales with video length unless aggressive keyframe subsampling is applied.

Windowed inference bounds memory usage by the window size rather than the sequence length. Each window creates a fresh cache, and since the cache is cleared between chunks, even hour-long videos maintain predictable GPU memory consumption. This makes the mode suitable for offline batch processing of arbitrarily long sequences.

Temporal Alignment and Stitching in Windowed Mode

Windowed inference requires explicit temporal alignment because each window produces predictions in its own coordinate frame.

After processing all windows, the model invokes _align_and_stitch_windows() (lines 668-756 in gct_stream_window.py). This method:

1. Estimates a similarity transform (scale, rotation, translation) from overlapping keyframes using _pairwise_alignment()
2. Warps one window's predictions into the previous window's coordinate frame via _warp_predictions()
3. Merges the aligned results seamlessly using _stitch_windows()

In contrast, streaming inference requires no explicit alignment because predictions are produced directly in a continuous temporal frame of reference.

Implementation Details and Key Entry Points

The implementation resides primarily in the GCTStream class within lingbot_map/models/gct_stream_window.py.

Method	Lines	Purpose
inference_streaming()	495-564	Processes continuous sequences with persistent caching
inference_windowed()	590-660	Handles chunked processing with cache reset per window
_align_and_stitch_windows()	668-756	Aligns and merges window predictions (windowed mode only)
clean_kv_cache()	Various	Resets the KV cache between windows

Both modes rely on the AggregatorStream (defined in lingbot_map/aggregator/stream.py) for the KV-cache backend and CameraCausalHead (in lingbot_map/heads/camera_head.py) for pose estimation.

Practical Code Examples

The following example demonstrates both inference modes using the same input tensor:

import torch
from lingbot_map.models.gct_stream_window import GCTStream

# Instantiate the model

model = GCTStream(pretrained_path="path/to/checkpoint.pt")
model.eval()

# Dummy video: 30 frames, 3×224×224 RGB

images = torch.rand(30, 3, 224, 224)  # shape [S, C, H, W]

# Streaming inference: single KV cache, continuous processing

stream_output = model.inference_streaming(
    images,
    num_scale_frames=4,
    keyframe_interval=2,
    output_device=torch.device("cpu")
)
print("Streaming pose shape:", stream_output["pose_enc"].shape)  # [1, 30, 9]

# Windowed inference: bounded memory with alignment

window_output = model.inference_windowed(
    images,
    window_size=12,
    overlap_keyframes=2,
    keyframe_interval=2,
    flow_threshold=0.0,
    output_device=torch.device("cpu")
)
print("Windowed pose shape:", window_output["pose_enc"].shape)  # [1, 30, 9]

In the streaming call, the model maintains one KV cache and processes frames sequentially. In the windowed call, the model repeatedly clears the cache, processes overlapping windows, and finally aligns the results using the overlap keyframes.

When to Use Each Mode

Choose streaming inference for real-time or online applications where video arrives frame-by-frame and you require continuously updated predictions with minimal latency. This mode works best when GPU memory can accommodate the full sequence or when using a high keyframe_interval to limit cache growth.

Choose windowed inference for offline processing of long videos where GPU memory is constrained. The windowed approach caps memory usage regardless of video length while maintaining high-quality predictions through overlap-based alignment.

Summary

• Streaming inference uses a single persistent KV cache for the entire sequence, processing frames continuously with natural temporal alignment but memory usage that scales with video length.
• Windowed inference splits sequences into independent chunks with separate KV caches, using clean_kv_cache() between windows to bound memory, then stitches results via _align_and_stitch_windows().
• Both modes share the same underlying architecture in GCTStream but differ in sequence handling: streaming processes the whole video at once (lines 495-564), while windowed processes chunks with overlap (lines 590-660).
• Streaming suits real-time applications; windowed suits memory-constrained batch processing of long sequences.

Frequently Asked Questions

What is the main memory advantage of windowed inference over streaming?

Windowed inference caps GPU memory usage at the size required for a single window plus overlap, regardless of video length. The inference_windowed() method calls clean_kv_cache() before processing each chunk, whereas streaming inference accumulates keyframes in a single cache that grows until it hits kv_cache_sliding_window.

How does windowed inference maintain temporal consistency between chunks?

The windowed mode uses overlap keyframes—the last frames of one window become the scale frames for the next window. After processing, _align_and_stitch_windows() estimates a similarity transform from these overlapping regions (via _pairwise_alignment()) and warps predictions into a common coordinate frame before merging.

Can I use flow-based keyframe selection in both modes?

Yes. Both inference_streaming() and inference_windowed() support flow-based keyframe selection via the flow_threshold parameter. In flow mode, the model computes optical flow between frames and either keeps the KV cache entry (if flow exceeds threshold) or rolls it back using _rollback_last_frame(). However, windowed mode applies this logic independently within each chunk.

Why does streaming inference not require explicit alignment?

Streaming inference maintains a single continuous KV cache and processes the entire sequence in one temporal reference frame. Since the model never resets its internal state (until explicitly cleared), predictions naturally align to the same coordinate system without requiring post-hoc stitching or warping operations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}