# Streaming vs Windowed Inference in LingBot-Map: Key Differences Explained

> Understand streaming vs windowed inference in LingBot-Map. Discover how each mode processes video data and how they differ for efficient content analysis.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: deep-dive
- Published: 2026-06-29

---

**Streaming inference maintains a single persistent KV cache for continuous frame-by-frame processing, while windowed inference partitions video into independent chunks with fresh caches and applies temporal alignment stitching to merge results.**

LingBot-Map (available in the `Robbyant/lingbot-map` repository) provides two distinct inference regimes through its **GCTStream** model architecture. Both modes leverage the same underlying KV-cache-enabled aggregator and causal camera head, but they differ fundamentally in sequence handling, memory management, and temporal alignment strategies.

## How Streaming Inference Processes Video Sequences

In **streaming inference**, the model processes the entire video sequence (or the portion that fits in GPU memory) within a single forward pass using one persistent KV cache.

The implementation in `inference_streaming` (lines 495–564 of [`lingbot_map/models/gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py)) first processes scale frames to establish the spatial reference, then iterates over remaining frames one-by-one. The KV cache grows as keyframes are stored, with non-keyframes optionally skipping cache append via the `_set_skip_append` mechanism to save memory. The cache can grow up to the configured `kv_cache_sliding_window` size, after which older entries are evicted.

**Keyframe selection** in streaming mode supports two strategies:

- **Fixed-interval**: Store KV entries every `keyframe_interval` frames
- **Flow-based**: Compute optical flow against the last keyframe using `flow_threshold`, deferring eviction and potentially rolling back the cache (`_rollback_last_frame`) if motion is insufficient

## How Windowed Inference Manages Memory and Alignment

**Windowed inference** addresses long-video memory constraints by splitting the sequence into independent windows, each receiving a fresh KV cache via `clean_kv_cache()` calls before processing.

The `inference_windowed` method (lines 590–660 of [`lingbot_map/models/gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py)) processes each window as a mini-streaming session, including its own scale frames and streaming frames. Critical to this approach is the **overlap mechanism**: each window shares `overlap_keyframes` with the next window, which become the subsequent window's scale frames. This overlap preserves bidirectional attention continuity across chunk boundaries.

After all windows complete, the model executes **temporal alignment stitching** through `_align_and_stitch_windows` (lines 668–756). This process:

1. Estimates similarity transforms (scale, rotation, translation) from overlapping keyframes using `_pairwise_alignment`
2. Warps predictions into the coordinate frame of the previous window via `_warp_predictions`
3. Merges results seamlessly using `_stitch_windows`

## Key Differences Between Streaming and Windowed Modes

| Aspect | Streaming Inference | Windowed Inference |
|--------|---------------------|--------------------|
| **KV Cache Scope** | Single cache persisting across all frames | Fresh cache per window, cleared via `clean_kv_cache()` |
| **Memory Footprint** | Scales with `kv_cache_sliding_window` and sequence length | Bounded by `window_size`, constant regardless of video length |
| **Temporal Alignment** | None required; native model coordinates | Post-processing alignment using `_align_and_stitch_windows` |
| **Keyframe Handling** | Fixed or flow-based selection within continuous stream | Fixed/flow-based plus overlap frames serving as next window's scale frames |
| **Typical Use Case** | Real-time/online processing with frame-by-frame arrival | Offline batch processing of long videos with memory constraints |

## Memory and Performance Implications

**Streaming mode** excels in real-time applications where frames arrive continuously and low latency is critical. However, memory usage grows with the number of stored keyframes unless you increase `keyframe_interval`, which may reduce temporal resolution.

**Windowed mode** guarantees predictable GPU memory usage even for hour-long videos, as memory is bounded by `window_size` rather than total frame count. The trade-off is computational overhead from the alignment stitching process and the need to process overlapping frames twice (once per adjacent window).

## Practical Implementation: Running Both Inference Modes

Both modes share the same `GCTStream` class from [`lingbot_map/models/gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py) and accept similar parameters for scale frames and keyframe selection.

```python
import torch
from lingbot_map.models.gct_stream_window import GCTStream

# Initialize model

model = GCTStream(pretrained_path="path/to/checkpoint.pt")
model.eval()

# Sample video tensor: 30 frames, 3×224×224 RGB

images = torch.rand(30, 3, 224, 224)  # Shape [S, C, H, W]

# Streaming inference: single KV cache, continuous processing

stream_output = model.inference_streaming(
    images,
    num_scale_frames=4,
    keyframe_interval=2,
    output_device=torch.device("cpu")
)
print("Streaming output shape:", stream_output["pose_enc"].shape)  # [1, 30, 9]

# Windowed inference: chunked processing with alignment

window_output = model.inference_windowed(
    images,
    window_size=12,
    overlap_keyframes=2,
    keyframe_interval=2,
    flow_threshold=0.0,
    output_device=torch.device("cpu")
)
print("Windowed output shape:", window_output["pose_enc"].shape)  # [1, 30, 9]

```

The streaming call exercises `inference_streaming` with its single persistent cache, while the windowed call repeatedly invokes `clean_kv_cache()` and finishes with `_align_and_stitch_windows` to fuse predictions.

## Summary

- **Streaming inference** uses one persistent KV cache for the entire sequence, ideal for real-time processing but memory scales with video length.
- **Windowed inference** splits video into chunks with independent caches, keeping memory bounded by `window_size` rather than sequence length.
- Windowed mode requires **temporal alignment stitching** (`_align_and_stitch_windows`) to merge overlapping window predictions into a coherent trajectory.
- Both modes rely on the shared `AggregatorStream` and `CameraCausalHead` architecture from [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py) and [`lingbot_map/heads/camera_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/camera_head.py).
- Keyframe selection (fixed-interval or flow-based) works in both modes, but windowed mode adds overlap frames to maintain temporal continuity across chunks.

## Frequently Asked Questions

### When should I use streaming inference over windowed inference?

Use **streaming inference** when processing live video streams or when you need continuous, low-latency predictions updated frame-by-frame. According to the LingBot-Map source code, this mode is optimal for online applications where the entire video is not available upfront and you want a single, continuously updated trajectory estimate without post-processing alignment.

### How does windowed inference maintain temporal consistency across chunks?

Windowed inference uses **overlapping keyframes** (`overlap_keyframes` parameter) where the final frames of one window become the scale frames of the next. After processing, the `_align_and_stitch_windows` method computes similarity transforms from these overlaps using `_pairwise_alignment`, then warps predictions into a common coordinate frame via `_warp_predictions` before final stitching.

### What is the maximum video length supported in streaming mode?

Streaming mode has no fixed frame limit, but maximum sequence length is constrained by available GPU memory and the `kv_cache_sliding_window` configuration. For very long videos (hundreds or thousands of frames), you must either increase `keyframe_interval` to skip more frames (reducing temporal resolution) or switch to windowed mode to maintain constant memory usage.

### Can I adjust the KV cache size in windowed inference?

In windowed inference, the effective KV cache size is determined by `window_size` (the number of keyframes per window) rather than a separate cache limit. Each window starts with a clean cache via `clean_kv_cache()`, ensuring memory usage stays predictable regardless of how you configure the sliding window parameter used in streaming mode.