# How Windowed Inference Works in LingBot-Map for Processing Very Long Sequences

> Discover how LingBot-Map's windowed inference processes long sequences with overlapping windows and fresh KV caches for constant GPU memory and global consistency. Learn more.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: internals
- Published: 2026-07-03

---

**Windowed inference in LingBot-Map processes arbitrarily long video sequences by dividing them into overlapping windows with fresh KV caches per window, enabling constant GPU memory usage while maintaining global consistency through scale-phase initialization and alignment-based stitching.**

The `Robbyant/lingbot-map` repository implements a sophisticated **windowed inference** pipeline that allows the model to handle video streams containing millions of frames without exhausting GPU memory. This approach keeps memory usage bounded to the size of a single window while producing globally consistent pose and depth predictions. The core implementation resides in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py), which orchestrates the division, processing, and stitching of video segments.

## The Windowed Inference Architecture

The pipeline treats long sequences as a series of manageable chunks rather than processing them monolithically. Each window contains a fixed number of **keyframes** stored in the KV cache plus additional **scale frames** processed bidirectionally to establish scene geometry.

### Window Construction and Overlap Management

The `inference_windowed` method (lines approximately 1200–1260 in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py)) constructs windows using the `window_size` parameter, which defaults to 16 keyframes. The actual frame count per window follows this calculation:

```python
actual_frames = scale_frames + (window_size - scale_frames) × keyframe_interval

```

Consecutive windows overlap by at least the number of scale frames (`overlap_size` defaults to `num_scale_frames`). Alternatively, you can specify `overlap_keyframes`, which the system converts to actual-frame overlap using the keyframe interval. This overlap ensures continuity between adjacent segments and enables the alignment phase to compute similarity transforms.

### Memory-Bounded KV Cache Strategy

Before processing each window, the model calls `clean_kv_cache()` to clear both the aggregator’s and camera head’s key-value caches. According to the source code in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py), this guarantees that GPU memory consumption never exceeds the requirements for a single window, regardless of total video length. The `AggregatorStream` class in [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py) manages this cache using either FlashInfer or SDPA backends, supporting eviction, rollback, and defer-eviction flags.

### Phase 1: Scale Initialization

The first `num_scale_frames` frames (default 8) of each window run together with `num_frame_per_block = num_scale_frames`. Because these frames share a **scale token**, they attend bidirectionally rather than causally. This yields a robust estimate of scene scale and improves downstream alignment quality. The scale phase establishes the coordinate framework before the streaming phase begins.

### Phase 2: Streaming with Keyframe Selection

After scale initialization, the remaining frames process one-by-one using two mutually exclusive modes:

**Fixed Interval Mode** (`keyframe_interval > 1`): The model retains KV cache entries only for every N-th frame (keyframes). Non-keyframes process with `skip_append=True` (via `_set_skip_append`), discarding their KV tensors immediately after the forward pass.

**Flow-Based Mode** (`flow_threshold > 0`): The system computes optical flow magnitude between the current frame and latest keyframe using `_compute_flow_magnitude`. If flow exceeds the threshold, the frame becomes a keyframe; otherwise, `_rollback_last_frame` reverses the operation after processing with deferred eviction (`_set_defer_eviction`).

The `CameraHead` in [`lingbot_map/heads/camera_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/camera_head.py) respects these KV-cache flags during causal pose refinement.

## Global Alignment and Stitching

After window processing, the system reconciles predictions across boundaries to ensure global consistency. The `_pairwise_alignment` method estimates similarity transforms (scale *s*, rotation *R*, translation *t*) using overlapping keyframes between adjacent windows. Each window then undergoes `_warp_predictions` to transform into the coordinate frame of the first window. Finally, `_stitch_windows` concatenates results while de-duplicating overlapping frames, producing a merged prediction dictionary containing full-length pose encodings, depth maps, and world-point clouds.

## Implementation Example

The following example demonstrates windowed inference on a 500-frame sequence:

```python
import torch
from lingbot_map.models.gct_stream_window_v2 import GCTStream

# Build the model (use a pretrained DINOv2 checkpoint if you have one)

model = GCTStream(
    img_size=518,
    patch_embed='dinov2_vitl14_reg',
    sliding_window_size=-1,          # full causal by default

    num_frame_for_scale=8,           # 8 scale frames per window

    enable_camera=True,
)

# Dummy video: 500 frames, 3×518×518 RGB, batch size 1

video = torch.rand(1, 500, 3, 518, 518)

# Run windowed inference:

preds = model.inference_windowed(
    images=video,
    window_size=16,          # 16 keyframes per window

    overlap_keyframes=4,     # 4-keyframe overlap → 8 actual-frame overlap

    keyframe_interval=2,     # store KV every other frame

    flow_threshold=0.0,      # disable flow-based mode

    output_device=torch.device('cpu')  # keep results on CPU

)

# `preds` now contains full-length pose_enc, depth, world_points, etc.

print(preds["pose_enc"].shape)   # → torch.Size([1, 500, 9])

```

## Key Source Files and Components

The windowed inference system relies on several specialized modules:

- **[`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py)**: Main implementation containing `inference_windowed`, KV-cache management, and alignment logic.
- **[`lingbot_map/models/gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py)**: Earlier version of the streaming window class.
- **[`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py)**: Implements `AggregatorStream` with FlashInfer/SDPA backends supporting cache eviction and rollback.
- **[`lingbot_map/heads/camera_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/camera_head.py)**: Causal camera head providing per-frame pose refinement respecting KV-cache flags.
- **[`lingbot_map/utils/pose_enc.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/pose_enc.py)**: Utilities for converting pose encodings to extrinsic/intrinsic matrices.
- **[`lingbot_map/utils/geometry.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/geometry.py)**: Contains `closed_form_inverse_se3` for inverting camera transforms during flow computation.

## Summary

- **Windowed inference** divides arbitrarily long sequences into overlapping windows with bounded GPU memory usage.
- Each window begins with a **scale phase** using bidirectional attention to establish robust scene geometry.
- The **streaming phase** processes remaining frames using either fixed keyframe intervals or optical flow thresholds to manage cache size.
- **Fresh KV caches** per window via `clean_kv_cache()` ensure memory consumption remains constant regardless of video length.
- **Global alignment** using `_pairwise_alignment` and `_warp_predictions` stitches windows into a consistent coordinate frame.
- The system supports **device offloading** via `output_device` to prevent OOM errors on extremely long sequences.

## Frequently Asked Questions

### How does windowed inference prevent GPU out-of-memory errors on long videos?

By calling `clean_kv_cache()` before processing each window, the system resets the KV cache to empty, ensuring GPU memory usage never exceeds the size required for a single window. This bounds memory consumption to O(window_size) rather than O(sequence_length), allowing processing of millions of frames.

### What is the difference between scale frames and keyframes in the windowed inference pipeline?

Scale frames are the initial frames in each window processed bidirectionally with shared scale tokens to establish scene geometry. Keyframes are selected frames (based on interval or optical flow) whose KV tensors are retained in the cache for causal attention during the streaming phase.

### How does the system maintain global consistency between separate windows?

After processing all windows, the system uses `_pairwise_alignment` to compute similarity transforms between overlapping regions, then warps predictions into a common coordinate frame using `_warp_predictions`. The `_stitch_windows` method merges results while removing duplicate overlapping frames.

### Can I use optical flow instead of fixed intervals for keyframe selection?

Yes, by setting `flow_threshold > 0` and `keyframe_interval=1`, the system enters flow-based mode. It computes optical flow magnitude between current and last keyframe frames using `_compute_flow_magnitude`, promoting frames to keyframes only when motion exceeds the threshold, and rolling back others via `_rollback_last_frame`.