# Streaming vs. Windowed Inference in LingBot-Map: How the Two Modes Differ

> Understand streaming vs windowed inference in LingBot Map. Learn how each mode processes video sequences with distinct KV cache management and alignment techniques for optimal performance.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: deep-dive
- Published: 2026-07-08

---

**Streaming inference processes video sequences in a single forward pass with one persistent KV cache, while windowed inference splits sequences into independent chunks with fresh KV caches per window and aligns results using overlapping keyframes.**

The **GCTStream** model in the [LingBot-Map](https://github.com/Robbyant/lingbot-map) repository provides two distinct inference regimes for visual localization: **streaming** and **windowed** inference modes. Both modes leverage the same underlying KV-cache-enabled aggregator and causal camera head architecture, but they differ fundamentally in how they manage memory, handle temporal sequences, and align predictions across time. Understanding these differences is crucial for optimizing real-time robotics applications versus offline batch processing workloads.

## How Streaming Inference Works in LingBot-Map

### Single KV Cache Architecture

In [`lingbot_map/models/gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py), the `inference_streaming` method (lines 495-564) processes the entire video sequence—or the portion that fits in GPU memory—in a single forward pass. The implementation first processes **scale frames** to establish the initial coordinate system, then iterates over subsequent frames one-by-one while reusing the **KV cache** across all timesteps.

The streaming mode maintains a **single KV cache** that persists for the entire inference run. As the model processes frames, it stores keyframes (or every frame when `keyframe_interval=1`) in the cache while optionally skipping cache append for non-keyframes using the `_set_skip_append` flag. The cache can grow up to the configured `kv_cache_sliding_window` size, which controls memory usage for long sequences.

### Keyframe Selection Strategies

Streaming inference supports two keyframe selection mechanisms:

- **Fixed-interval keyframes**: Controlled by the `keyframe_interval` parameter, which determines how often to update the KV cache
- **Flow-based keyframes**: When using `flow_threshold`, the model defers eviction, runs a forward pass to compute optical flow, and either keeps the KV entry for keyframes or rolls back the cache using `_rollback_last_frame` for non-keyframes

This approach produces predictions directly in the model's temporal frame of reference without explicit alignment steps, making it ideal for **real-time or online processing** where video streams arrive frame-by-frame.

## How Windowed Inference Works in LingBot-Map

### Chunked Processing with Independent Caches

The `inference_windowed` method (lines 590-660) implements a divide-and-conquer strategy that splits long sequences into manageable **windows**. Before processing each window, the model calls `clean_kv_cache()` to create a fresh cache, ensuring that memory usage remains bounded by the window size rather than the entire video length.

Each window processes its own scale frames followed by its streaming frames independently. The architecture introduces **overlap frames**—typically the last few keyframes of a window—that become the next window's scale frames. This overlap ensures that bidirectional attention from scale frames is preserved across window boundaries.

### Temporal Alignment and Stitching

Unlike streaming mode, windowed inference requires explicit **temporal alignment** after processing. The `_align_and_stitch_windows` method (lines 668-756) handles this by:

1. Estimating a **similarity transform** (scale, rotation, translation) from overlapping keyframes using `_pairwise_alignment`
2. Warping one window's predictions into the coordinate frame of the previous window via `_warp_predictions`
3. Merging results seamlessly using `_stitch_windows`

This alignment step allows windowed inference to maintain global consistency despite processing chunks independently, making it suitable for **offline batch processing** of long videos where GPU memory constraints would otherwise limit processing.

## Memory and Performance Comparison

### Memory Footprint Characteristics

**Streaming inference** scales its memory usage with the number of frames kept in the KV cache, controlled by `kv_cache_sliding_window`. For very long videos, the cache can become substantial unless a high `keyframe_interval` is used to limit stored states.

**Windowed inference** provides predictable memory bounds determined solely by the `window_size` parameter. Even for hour-long sequences, GPU memory usage stays constant because the KV cache is cleared after each chunk.

### Temporal Alignment Requirements

- **Streaming mode** requires no explicit alignment—predictions emerge naturally from the continuous KV cache state
- **Windowed mode** requires the `_align_and_stitch_windows` pipeline to merge predictions from independent chunks, adding computational overhead but enabling parallel processing of windows

## Practical Implementation: Code Examples

The following example demonstrates both inference modes using the `GCTStream` class:

```python
import torch
from lingbot_map.models.gct_stream_window import GCTStream

# Instantiate the model (use a pretrained checkpoint if needed)

model = GCTStream(pretrained_path="path/to/checkpoint.pt")
model.eval()

# Dummy video: 30 frames, 3×224×224 RGB, values in [0, 1]

images = torch.rand(30, 3, 224, 224)          # shape [S, C, H, W]

# ------------------------------------------------------------------

# 1️⃣ Streaming inference (single KV cache)

# ------------------------------------------------------------------

stream_output = model.inference_streaming(
    images,                     # can also be B×S×C×H×W

    num_scale_frames=4,         # first 4 frames are used for scale estimation

    keyframe_interval=2,        # store KV every 2nd frame to save memory

    output_device=torch.device("cpu")   # off‑load predictions to CPU

)

print("Streaming pose shape:", stream_output["pose_enc"].shape)   # → [1, 30, 9]

# ------------------------------------------------------------------

# 2️⃣ Windowed inference (bounded memory, alignment)

# ------------------------------------------------------------------

window_output = model.inference_windowed(
    images,
    window_size=12,            # 12 keyframes per window (including scale frames)

    overlap_keyframes=2,       # 2 keyframes overlap between windows

    keyframe_interval=2,       # same keyframe spacing inside each window

    flow_threshold=0.0,        # use fixed‑interval keyframes (no flow mode)

    output_device=torch.device("cpu")
)

print("Windowed pose shape:", window_output["pose_enc"].shape)   # → [1, 30, 9]

```

In the streaming call, the model maintains a single KV cache throughout the entire sequence. In the windowed call, the model repeatedly invokes `clean_kv_cache()` between windows and finally calls `_align_and_stitch_windows` to fuse predictions.

## Core Architecture and Source Files

Both inference modes rely on shared components from the LingBot-Map architecture:

- **[`lingbot_map/models/gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py)**: Implements both `inference_streaming` and `inference_windowed`, plus KV-cache utilities including `_set_skip_append`, `_rollback_last_frame`, and the alignment helpers (lines 495-756)
- **[`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py)**: Defines `AggregatorStream` providing the KV-cache backend supporting FlashInfer or SDPA attention mechanisms
- **[`lingbot_map/heads/camera_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/camera_head.py)**: Implements the `CameraCausalHead` used by both modes for pose estimation
- **[`lingbot_map/models/gct_base.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_base.py)**: Base class containing common layers shared between streaming and windowed implementations

## Summary

- **Streaming inference** uses a single persistent KV cache for the entire sequence, processing frames incrementally with optional keyframe selection via fixed intervals or optical flow thresholds.
- **Windowed inference** partitions videos into independent chunks with fresh KV caches per window, using overlapping keyframes to maintain temporal consistency across boundaries.
- **Memory constraints** favor windowed mode for long videos, while latency requirements favor streaming mode for real-time applications.
- **Alignment complexity** differs significantly: streaming requires no post-processing, while windowed mode requires `_align_and_stitch_windows` to merge chunk predictions using similarity transforms estimated from overlap regions.

## Frequently Asked Questions

### What is the memory impact of streaming inference versus windowed inference?

Streaming inference memory scales with the total number of keyframes stored in the KV cache, potentially growing large for long sequences unless limited by `kv_cache_sliding_window`. Windowed inference bounds memory usage to the size of a single window regardless of video length, as implemented by calling `clean_kv_cache()` before processing each chunk in `inference_windowed`.

### When should I use windowed inference instead of streaming inference?

Use **windowed inference** when processing long videos offline where GPU memory is constrained or when you need to process videos longer than the available VRAM can handle with a single KV cache. Use **streaming inference** for real-time applications where video arrives frame-by-frame and you require continuous, low-latency predictions without alignment overhead.

### How does temporal alignment work in windowed inference mode?

After processing each window independently, the `_align_and_stitch_windows` method (lines 668-756) aligns predictions by computing a similarity transform between overlapping keyframes from adjacent windows using `_pairwise_alignment`. It then warps the predictions into a common coordinate frame via `_warp_predictions` and merges them using `_stitch_windows`, ensuring global temporal consistency despite independent chunk processing.

### Can I use flow-based keyframe selection in both inference modes?

Yes, both modes support flow-based keyframe selection via the `flow_threshold` parameter. In streaming mode, the model uses `_rollback_last_frame` to exclude non-keyframes from the persistent cache. In windowed mode, the same logic applies within each window, but the overlap frames between windows ensure that scale frame bidirectional attention is preserved regardless of which frames are selected as keyframes within the window.