Streaming vs. Windowed Inference in LingBot-Map: How the Two Modes Differ

Streaming inference processes video sequences in a single forward pass with one persistent KV cache, while windowed inference splits sequences into independent chunks with fresh KV caches per window and aligns results using overlapping keyframes.

The GCTStream model in the LingBot-Map repository provides two distinct inference regimes for visual localization: streaming and windowed inference modes. Both modes leverage the same underlying KV-cache-enabled aggregator and causal camera head architecture, but they differ fundamentally in how they manage memory, handle temporal sequences, and align predictions across time. Understanding these differences is crucial for optimizing real-time robotics applications versus offline batch processing workloads.

How Streaming Inference Works in LingBot-Map

Single KV Cache Architecture

In lingbot_map/models/gct_stream_window.py, the inference_streaming method (lines 495-564) processes the entire video sequence—or the portion that fits in GPU memory—in a single forward pass. The implementation first processes scale frames to establish the initial coordinate system, then iterates over subsequent frames one-by-one while reusing the KV cache across all timesteps.

The streaming mode maintains a single KV cache that persists for the entire inference run. As the model processes frames, it stores keyframes (or every frame when keyframe_interval=1) in the cache while optionally skipping cache append for non-keyframes using the _set_skip_append flag. The cache can grow up to the configured kv_cache_sliding_window size, which controls memory usage for long sequences.

Keyframe Selection Strategies

Streaming inference supports two keyframe selection mechanisms:

• Fixed-interval keyframes: Controlled by the keyframe_interval parameter, which determines how often to update the KV cache
• Flow-based keyframes: When using flow_threshold, the model defers eviction, runs a forward pass to compute optical flow, and either keeps the KV entry for keyframes or rolls back the cache using _rollback_last_frame for non-keyframes

This approach produces predictions directly in the model's temporal frame of reference without explicit alignment steps, making it ideal for real-time or online processing where video streams arrive frame-by-frame.

How Windowed Inference Works in LingBot-Map

Chunked Processing with Independent Caches

The inference_windowed method (lines 590-660) implements a divide-and-conquer strategy that splits long sequences into manageable windows. Before processing each window, the model calls clean_kv_cache() to create a fresh cache, ensuring that memory usage remains bounded by the window size rather than the entire video length.

Each window processes its own scale frames followed by its streaming frames independently. The architecture introduces overlap frames—typically the last few keyframes of a window—that become the next window's scale frames. This overlap ensures that bidirectional attention from scale frames is preserved across window boundaries.

Temporal Alignment and Stitching

Unlike streaming mode, windowed inference requires explicit temporal alignment after processing. The _align_and_stitch_windows method (lines 668-756) handles this by:

1. Estimating a similarity transform (scale, rotation, translation) from overlapping keyframes using _pairwise_alignment
2. Warping one window's predictions into the coordinate frame of the previous window via _warp_predictions
3. Merging results seamlessly using _stitch_windows

This alignment step allows windowed inference to maintain global consistency despite processing chunks independently, making it suitable for offline batch processing of long videos where GPU memory constraints would otherwise limit processing.

Memory and Performance Comparison

Memory Footprint Characteristics

Streaming inference scales its memory usage with the number of frames kept in the KV cache, controlled by kv_cache_sliding_window. For very long videos, the cache can become substantial unless a high keyframe_interval is used to limit stored states.

Windowed inference provides predictable memory bounds determined solely by the window_size parameter. Even for hour-long sequences, GPU memory usage stays constant because the KV cache is cleared after each chunk.

Temporal Alignment Requirements

• Streaming mode requires no explicit alignment—predictions emerge naturally from the continuous KV cache state
• Windowed mode requires the _align_and_stitch_windows pipeline to merge predictions from independent chunks, adding computational overhead but enabling parallel processing of windows

Practical Implementation: Code Examples

The following example demonstrates both inference modes using the GCTStream class:

import torch
from lingbot_map.models.gct_stream_window import GCTStream

# Instantiate the model (use a pretrained checkpoint if needed)

model = GCTStream(pretrained_path="path/to/checkpoint.pt")
model.eval()

# Dummy video: 30 frames, 3×224×224 RGB, values in [0, 1]

images = torch.rand(30, 3, 224, 224)          # shape [S, C, H, W]

# ------------------------------------------------------------------

# 1️⃣ Streaming inference (single KV cache)

# ------------------------------------------------------------------

stream_output = model.inference_streaming(
    images,                     # can also be B×S×C×H×W

    num_scale_frames=4,         # first 4 frames are used for scale estimation

    keyframe_interval=2,        # store KV every 2nd frame to save memory

    output_device=torch.device("cpu")   # off‑load predictions to CPU

)

print("Streaming pose shape:", stream_output["pose_enc"].shape)   # → [1, 30, 9]

# ------------------------------------------------------------------

# 2️⃣ Windowed inference (bounded memory, alignment)

# ------------------------------------------------------------------

window_output = model.inference_windowed(
    images,
    window_size=12,            # 12 keyframes per window (including scale frames)

    overlap_keyframes=2,       # 2 keyframes overlap between windows

    keyframe_interval=2,       # same keyframe spacing inside each window

    flow_threshold=0.0,        # use fixed‑interval keyframes (no flow mode)

    output_device=torch.device("cpu")
)

print("Windowed pose shape:", window_output["pose_enc"].shape)   # → [1, 30, 9]

In the streaming call, the model maintains a single KV cache throughout the entire sequence. In the windowed call, the model repeatedly invokes clean_kv_cache() between windows and finally calls _align_and_stitch_windows to fuse predictions.

Core Architecture and Source Files

Both inference modes rely on shared components from the LingBot-Map architecture:

• lingbot_map/models/gct_stream_window.py: Implements both inference_streaming and inference_windowed, plus KV-cache utilities including _set_skip_append, _rollback_last_frame, and the alignment helpers (lines 495-756)
• lingbot_map/aggregator/stream.py: Defines AggregatorStream providing the KV-cache backend supporting FlashInfer or SDPA attention mechanisms
• lingbot_map/heads/camera_head.py: Implements the CameraCausalHead used by both modes for pose estimation
• lingbot_map/models/gct_base.py: Base class containing common layers shared between streaming and windowed implementations

Summary

• Streaming inference uses a single persistent KV cache for the entire sequence, processing frames incrementally with optional keyframe selection via fixed intervals or optical flow thresholds.
• Windowed inference partitions videos into independent chunks with fresh KV caches per window, using overlapping keyframes to maintain temporal consistency across boundaries.
• Memory constraints favor windowed mode for long videos, while latency requirements favor streaming mode for real-time applications.
• Alignment complexity differs significantly: streaming requires no post-processing, while windowed mode requires _align_and_stitch_windows to merge chunk predictions using similarity transforms estimated from overlap regions.

Frequently Asked Questions

What is the memory impact of streaming inference versus windowed inference?

Streaming inference memory scales with the total number of keyframes stored in the KV cache, potentially growing large for long sequences unless limited by kv_cache_sliding_window. Windowed inference bounds memory usage to the size of a single window regardless of video length, as implemented by calling clean_kv_cache() before processing each chunk in inference_windowed.

When should I use windowed inference instead of streaming inference?

Use windowed inference when processing long videos offline where GPU memory is constrained or when you need to process videos longer than the available VRAM can handle with a single KV cache. Use streaming inference for real-time applications where video arrives frame-by-frame and you require continuous, low-latency predictions without alignment overhead.

How does temporal alignment work in windowed inference mode?

After processing each window independently, the _align_and_stitch_windows method (lines 668-756) aligns predictions by computing a similarity transform between overlapping keyframes from adjacent windows using _pairwise_alignment. It then warps the predictions into a common coordinate frame via _warp_predictions and merges them using _stitch_windows, ensuring global temporal consistency despite independent chunk processing.

Can I use flow-based keyframe selection in both inference modes?

Yes, both modes support flow-based keyframe selection via the flow_threshold parameter. In streaming mode, the model uses _rollback_last_frame to exclude non-keyframes from the persistent cache. In windowed mode, the same logic applies within each window, but the overlap frames between windows ensure that scale frame bidirectional attention is preserved regardless of which frames are selected as keyframes within the window.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}