# Streaming vs Windowed Inference in GCTStream: A Complete Technical Guide

> Understand streaming vs windowed inference in GCTStream. Explore low-latency real-time SLAM with persistent KV-cache or bounded memory with overlapping windows for efficient processing.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: how-to-guide
- Published: 2026-07-04

---

**Streaming inference processes entire video sequences with a persistent KV-cache for low-latency real-time SLAM, while windowed inference splits long sequences into overlapping windows with fresh caches and geometric alignment to maintain bounded memory usage.**

`GCTStream` in the `Robbyant/lingbot-map` repository implements two distinct inference pipelines for causal transformer-based depth and pose estimation. Both modes leverage a **KV-cache** (key-value cache) for efficient online processing, but differ fundamentally in how they manage memory and handle long sequences. Understanding these differences is critical for deploying the model on anything from real-time robotics to offline reconstruction of hours-long RGB-D scans.

## Architecture Overview

At the core of both pipelines is a causal transformer that maintains a **KV-cache** to avoid recomputing attention for past frames. The class definition begins at line 74 in [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py), with the actual inference implementations residing in separate methods. The key architectural distinction lies in cache lifetime: streaming maintains a single cache for the entire sequence, while windowed periodically resets the cache and stitches results using geometric alignment.

## Streaming Inference

Streaming inference is designed for real-time applications where the entire video is processed sequentially in a single pass.

### Processing Scope

In `inference_streaming` (lines 490-545 of [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py)), the model runs over the **entire video sequence** with the KV-cache kept alive throughout. Each new frame can attend to all previously cached tokens, enabling true causal online processing. The cache is initialized once via `clean_kv_cache()` at the sequence start and persists until completion.

### Memory Dynamics

Memory consumption grows with the number of **keyframes** stored in the cache. For long videos, the cache can become prohibitively large unless keyframe subsampling is employed. The growth is controlled by two mechanisms:

- **Fixed-interval mode**: Set `keyframe_interval > 1` to store only every Nth frame
- **Flow-based mode**: Set `flow_threshold > 0` to dynamically select keyframes based on optical flow magnitude

### Keyframe Handling

During non-keyframe inference, the cache temporarily disables appending via `_set_skip_append`. In flow-based mode, the system uses `_set_defer_eviction`, `_rollback_last_frame`, and `_execute_deferred_eviction` (lines 300-329) to tentatively keep frames in the cache and undo them if they fail the keyframe criteria.

### Implementation Location

The streaming pipeline relies on [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py) for the underlying `AggregatorStream` module, which manages the FlashInfer or SDPA-backed KV-cache operations.

## Windowed Inference

Windowed inference trades unbounded memory growth for computational overhead by processing long sequences in chunks.

### Processing Strategy

The `inference_windowed` method (lines 558-587) splits the input into a series of **overlapping windows**. Each window receives a **fresh KV-cache**; `clean_kv_cache()` is called at the start of every window, ensuring memory usage never exceeds the capacity required for a single window. This design makes the approach safe for arbitrarily long videos, such as hours-long RGB-D scans.

### Memory Constraints

Unlike streaming, memory is **bounded by the window size** rather than the sequence length. The cache never contains more than the number of keyframes inside one window, regardless of total video duration. This predictability is essential for batch processing large datasets on hardware with limited VRAM.

### Alignment and Stitching

Windowed inference introduces **scale frames** (bidirectional attention) that are re-processed at the start of every window to anchor depth scaling. To maintain global consistency, the pipeline uses:

- `_stitch_windows` for combining per-window predictions
- `_pairwise_alignment` for geometric alignment between adjacent windows
- `_warp_predictions` for transforming predictions into a common coordinate frame
- `_align_and_stitch_windows` (lines 671-745) for the final global alignment

The overlap regions between windows contain the scale frames for the next window, enabling smooth alignment via the KV-cache of the previous window and the scale frames of the next.

## Key Differences at a Glance

| Aspect | Streaming Inference | Windowed Inference |
|--------|---------------------|-------------------|
| **Cache Lifecycle** | Single cache for entire sequence; cleared once | Fresh cache per window; cleared repeatedly |
| **Memory Usage** | Grows with keyframe count; unbounded for long videos | Bounded by window size; constant regardless of video length |
| **Attention Pattern** | Full causal attention to all past keyframes | Causal within window; scale frames get bidirectional attention |
| **Output Handling** | Single concatenated tensor | Per-window predictions aligned and stitched with geometric transforms |
| **Typical Use Case** | Real-time SLAM requiring low latency | Offline processing of very long sequences with memory constraints |

## Code Examples

### Streaming Inference Example

```python
import torch
from lingbot_map.models.gct_stream import GCTStream

# Build model with streaming enabled (default)

model = GCTStream(enable_stream_inference=True)
model.eval().cuda()

# Dummy video: batch=1, 30 frames, RGB 518×518

images = torch.rand(1, 30, 3, 518, 518, device='cuda')

# Run streaming inference; keep every frame as keyframe

preds = model.inference_streaming(
    images,
    num_scale_frames=4,      # First 4 frames are scale frames

    keyframe_interval=1,     # Every frame stays in KV-cache

)

print(preds["pose_enc"].shape)   # → torch.Size([1, 30, 9])

```

### Windowed Inference Example

```python
import torch
from lingbot_map.models.gct_stream import GCTStream

model = GCTStream(enable_stream_inference=True)
model.eval().cuda()

# Very long video: 2000 frames

images = torch.rand(1, 2000, 3, 518, 518, device='cuda')

# Run windowed inference with bounded memory

preds = model.inference_windowed(
    images,
    window_size=16,           # 16 keyframes per window

    overlap_keyframes=2,      # Overlap for alignment

    keyframe_interval=4,      # Store every 4th frame only

    flow_threshold=0.0,       # Use fixed-interval keyframes

)

print(preds["pose_enc"].shape)        # → torch.Size([1, 2000, 9])

print(preds["alignment_mode"])        # → "scaled"

```

## Summary

- **Streaming inference** (`inference_streaming`) maintains a persistent KV-cache across the entire video, enabling low-latency processing but requiring memory management via keyframe intervals for long sequences.
- **Windowed inference** (`inference_windowed`) processes video in overlapping chunks with per-window cache resets, using geometric alignment (`_stitch_windows`, `_pairwise_alignment`) to maintain global consistency while keeping memory bounded.
- Both methods support **keyframe selection** via fixed intervals or optical flow thresholds, but only windowed inference requires **scale frame reprocessing** and alignment stitching.
- The implementations reside in [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py) (lines 490-545 for streaming, lines 558-587 for windowed), with cache utilities at lines 300-329 and stitching logic at lines 671-745.

## Frequently Asked Questions

### When should I use streaming inference versus windowed inference?

Use **streaming inference** for real-time SLAM applications where low latency is critical and you can afford the memory growth, or when processing shorter clips where the KV-cache remains manageable. Use **windowed inference** for offline processing of very long sequences (hours-long scans) where memory constraints dominate and you need guaranteed bounded VRAM usage regardless of video length.

### How does windowed inference maintain geometric consistency across window boundaries?

Windowed inference uses **overlap frames** containing the scale frames from adjacent windows, then applies `_pairwise_alignment` and `_warp_predictions` to geometrically align predictions before stitching. The `_align_and_stitch_windows` method computes relative transformations (scale, rotation, translation) between overlapping regions to ensure the final trajectory remains globally consistent despite processing chunks independently.

### What controls memory usage in streaming inference mode?

Memory in streaming mode is controlled primarily by the **keyframe interval**. Setting `keyframe_interval > 1` reduces cache size by only storing every Nth frame. Alternatively, enabling `flow_threshold > 0` activates optical-flow-based keyframe selection, where the system uses `_set_defer_eviction` and `_rollback_last_frame` to dynamically decide whether to retain frames based on motion magnitude.

### Why does windowed inference reprocess scale frames at the start of each window?

Scale frames receive **bidirectional attention** (unlike the causal attention used for other frames) to establish absolute scale references. Reprocessing them at each window start provides a fresh anchor for depth prediction, preventing drift that would otherwise accumulate if the model relied solely on the previous window's cached state. This reprocessing occurs before the causal KV-cache is populated for the current window.