Streaming vs Windowed Inference in GCTStream: A Complete Technical Guide

Streaming inference processes entire video sequences with a persistent KV-cache for low-latency real-time SLAM, while windowed inference splits long sequences into overlapping windows with fresh caches and geometric alignment to maintain bounded memory usage.

GCTStream in the Robbyant/lingbot-map repository implements two distinct inference pipelines for causal transformer-based depth and pose estimation. Both modes leverage a KV-cache (key-value cache) for efficient online processing, but differ fundamentally in how they manage memory and handle long sequences. Understanding these differences is critical for deploying the model on anything from real-time robotics to offline reconstruction of hours-long RGB-D scans.

Architecture Overview

At the core of both pipelines is a causal transformer that maintains a KV-cache to avoid recomputing attention for past frames. The class definition begins at line 74 in lingbot_map/models/gct_stream.py, with the actual inference implementations residing in separate methods. The key architectural distinction lies in cache lifetime: streaming maintains a single cache for the entire sequence, while windowed periodically resets the cache and stitches results using geometric alignment.

Streaming Inference

Streaming inference is designed for real-time applications where the entire video is processed sequentially in a single pass.

Processing Scope

In inference_streaming (lines 490-545 of lingbot_map/models/gct_stream.py), the model runs over the entire video sequence with the KV-cache kept alive throughout. Each new frame can attend to all previously cached tokens, enabling true causal online processing. The cache is initialized once via clean_kv_cache() at the sequence start and persists until completion.

Memory Dynamics

Memory consumption grows with the number of keyframes stored in the cache. For long videos, the cache can become prohibitively large unless keyframe subsampling is employed. The growth is controlled by two mechanisms:

• Fixed-interval mode: Set keyframe_interval > 1 to store only every Nth frame
• Flow-based mode: Set flow_threshold > 0 to dynamically select keyframes based on optical flow magnitude

Keyframe Handling

During non-keyframe inference, the cache temporarily disables appending via _set_skip_append. In flow-based mode, the system uses _set_defer_eviction, _rollback_last_frame, and _execute_deferred_eviction (lines 300-329) to tentatively keep frames in the cache and undo them if they fail the keyframe criteria.

Implementation Location

The streaming pipeline relies on lingbot_map/aggregator/stream.py for the underlying AggregatorStream module, which manages the FlashInfer or SDPA-backed KV-cache operations.

Windowed Inference

Windowed inference trades unbounded memory growth for computational overhead by processing long sequences in chunks.

Processing Strategy

The inference_windowed method (lines 558-587) splits the input into a series of overlapping windows. Each window receives a fresh KV-cache; clean_kv_cache() is called at the start of every window, ensuring memory usage never exceeds the capacity required for a single window. This design makes the approach safe for arbitrarily long videos, such as hours-long RGB-D scans.

Memory Constraints

Unlike streaming, memory is bounded by the window size rather than the sequence length. The cache never contains more than the number of keyframes inside one window, regardless of total video duration. This predictability is essential for batch processing large datasets on hardware with limited VRAM.

Alignment and Stitching

Windowed inference introduces scale frames (bidirectional attention) that are re-processed at the start of every window to anchor depth scaling. To maintain global consistency, the pipeline uses:

• _stitch_windows for combining per-window predictions
• _pairwise_alignment for geometric alignment between adjacent windows
• _warp_predictions for transforming predictions into a common coordinate frame
• _align_and_stitch_windows (lines 671-745) for the final global alignment

The overlap regions between windows contain the scale frames for the next window, enabling smooth alignment via the KV-cache of the previous window and the scale frames of the next.

Key Differences at a Glance

Aspect	Streaming Inference	Windowed Inference
Cache Lifecycle	Single cache for entire sequence; cleared once	Fresh cache per window; cleared repeatedly
Memory Usage	Grows with keyframe count; unbounded for long videos	Bounded by window size; constant regardless of video length
Attention Pattern	Full causal attention to all past keyframes	Causal within window; scale frames get bidirectional attention
Output Handling	Single concatenated tensor	Per-window predictions aligned and stitched with geometric transforms
Typical Use Case	Real-time SLAM requiring low latency	Offline processing of very long sequences with memory constraints

Code Examples

Streaming Inference Example

import torch
from lingbot_map.models.gct_stream import GCTStream

# Build model with streaming enabled (default)

model = GCTStream(enable_stream_inference=True)
model.eval().cuda()

# Dummy video: batch=1, 30 frames, RGB 518×518

images = torch.rand(1, 30, 3, 518, 518, device='cuda')

# Run streaming inference; keep every frame as keyframe

preds = model.inference_streaming(
    images,
    num_scale_frames=4,      # First 4 frames are scale frames

    keyframe_interval=1,     # Every frame stays in KV-cache

)

print(preds["pose_enc"].shape)   # → torch.Size([1, 30, 9])

Windowed Inference Example

import torch
from lingbot_map.models.gct_stream import GCTStream

model = GCTStream(enable_stream_inference=True)
model.eval().cuda()

# Very long video: 2000 frames

images = torch.rand(1, 2000, 3, 518, 518, device='cuda')

# Run windowed inference with bounded memory

preds = model.inference_windowed(
    images,
    window_size=16,           # 16 keyframes per window

    overlap_keyframes=2,      # Overlap for alignment

    keyframe_interval=4,      # Store every 4th frame only

    flow_threshold=0.0,       # Use fixed-interval keyframes

)

print(preds["pose_enc"].shape)        # → torch.Size([1, 2000, 9])

print(preds["alignment_mode"])        # → "scaled"

Summary

• Streaming inference (inference_streaming) maintains a persistent KV-cache across the entire video, enabling low-latency processing but requiring memory management via keyframe intervals for long sequences.
• Windowed inference (inference_windowed) processes video in overlapping chunks with per-window cache resets, using geometric alignment (_stitch_windows, _pairwise_alignment) to maintain global consistency while keeping memory bounded.
• Both methods support keyframe selection via fixed intervals or optical flow thresholds, but only windowed inference requires scale frame reprocessing and alignment stitching.
• The implementations reside in lingbot_map/models/gct_stream.py (lines 490-545 for streaming, lines 558-587 for windowed), with cache utilities at lines 300-329 and stitching logic at lines 671-745.

Frequently Asked Questions

When should I use streaming inference versus windowed inference?

Use streaming inference for real-time SLAM applications where low latency is critical and you can afford the memory growth, or when processing shorter clips where the KV-cache remains manageable. Use windowed inference for offline processing of very long sequences (hours-long scans) where memory constraints dominate and you need guaranteed bounded VRAM usage regardless of video length.

How does windowed inference maintain geometric consistency across window boundaries?

Windowed inference uses overlap frames containing the scale frames from adjacent windows, then applies _pairwise_alignment and _warp_predictions to geometrically align predictions before stitching. The _align_and_stitch_windows method computes relative transformations (scale, rotation, translation) between overlapping regions to ensure the final trajectory remains globally consistent despite processing chunks independently.

What controls memory usage in streaming inference mode?

Memory in streaming mode is controlled primarily by the keyframe interval. Setting keyframe_interval > 1 reduces cache size by only storing every Nth frame. Alternatively, enabling flow_threshold > 0 activates optical-flow-based keyframe selection, where the system uses _set_defer_eviction and _rollback_last_frame to dynamically decide whether to retain frames based on motion magnitude.

Why does windowed inference reprocess scale frames at the start of each window?

Scale frames receive bidirectional attention (unlike the causal attention used for other frames) to establish absolute scale references. Reprocessing them at each window start provides a fresh anchor for depth prediction, preventing drift that would otherwise accumulate if the model relied solely on the previous window's cached state. This reprocessing occurs before the causal KV-cache is populated for the current window.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}