# How to Implement Windowed Inference Mode for Video Sequences Longer Than 3000 Frames

> Implement windowed inference for long videos using GCTStream.inference_windowed. Process videos over 3000 frames without memory errors by splitting into windows, resetting KV cache, selecting keyframes, and stitching results.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: how-to-guide
- Published: 2026-07-21

---

**To process videos longer than 3000 frames without GPU out-of-memory errors, call `GCTStream.inference_windowed()` in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py), which splits the input into overlapping windows, resets the KV cache per window, dynamically selects keyframes, and stitches results with a similarity transform estimated from shared overlap frames.**

Ling-Bot-Map, available at `Robbyant/lingbot-map`, is an open-source spatial-reasoning system that normally processes video with causal attention and a KV cache. While effective for short clips, this cache grows linearly with frame count and exhausts GPU memory on long sequences. Implementing **windowed inference mode for video sequences** bounds memory usage to a single window, enabling inference on 10,000-frame videos or longer.

## Why Standard Streaming Inference Fails Beyond 3000 Frames

Standard causal streaming retains every keyframe in the KV cache so that new frames can attend to all prior context. On a 5,000- or 10,000-frame video, this unbounded cache growth triggers an out-of-memory error regardless of how small the batch size is. The repository solves this by providing a windowed alternative that resets the cache at regular intervals while preserving geometric continuity across segments.

## Core Architecture of Windowed Inference

The windowed strategy implemented in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py) rests on four pillars that keep memory constant regardless of total video length.

**Overlapping windows with a fresh KV cache** — The video is divided into windows of `window_size` keyframes. Before each window begins, `self.clean_kv_cache()` wipes the previous context, ensuring the cache never exceeds `kv_cache_sliding_window` entries. An overlap region between consecutive windows provides common frames for later geometric alignment.

**Bidirectional scale frames** — Every window starts with a small set of scale frames that are processed together. These frames receive a full forward pass with `num_frame_per_block=window_scale` to create a shared spatial reference. The rest of the window then proceeds with standard causal streaming.

**Keyframe selection** — Only selected frames populate the KV cache. The model supports two modes:

- **Fixed-interval mode**: A frame is stored every `keyframe_interval` steps. Non-keyframes call `self._set_skip_append(True)` to bypass cache insertion.
- **Flow-based mode**: When `flow_threshold > 0`, the model defers eviction with `self._set_defer_eviction(True)`, computes optical flow magnitude via `_compute_flow_magnitude()`, and promotes high-motion frames to keyframes. The `max_non_keyframe_gap` parameter forces a keyframe after long static stretches.

**Cross-window alignment and stitching** — After all windows finish, the pipeline calls `_align_and_stitch_windows()`. Internally, `_pairwise_alignment()` estimates a similarity transform (scale, rotation, translation) from overlapping keyframes; `_warp_predictions()` applies it; and `_stitch_windows()` concatenates the sequences while discarding duplicated overlap frames.

## Inside the `inference_windowed()` Pipeline

The following walkthrough traces the exact operations performed by `GCTStream.inference_windowed()` in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py).

### Input Normalization and Overlap Setup

At lines 1024–1028, the method ensures a batch dimension exists:

```python
if len(images.shape) == 4:
    images = images.unsqueeze(0)

```

Between lines 1093–1101, the overlap is resolved. If `overlap_keyframes` is provided, it is converted to a raw frame count using `keyframe_interval`. If `overlap_size` is provided, it is used directly. Otherwise, the effective overlap defaults to the window size.

### Phase 1 – Processing Scale Frames

Lines 1159–1170 process the scale frames to obtain bidirectional context before causal streaming starts:

```python
scale_out = self.forward(
    ...,
    num_frame_per_block=window_scale,
    causal_inference=True
)

```

### Phase 2 – Streaming Frames and Keyframe Detection

For the remaining frames in the window, the model switches to causal one-by-one processing.

If flow-based keyframe detection is enabled (`flow_threshold > 0`), the logic at lines 1196–1229 performs the following:

1. Defer cache eviction with `self._set_defer_eviction(True)`.
2. Run the forward pass.
3. Compute optical flow magnitude using `_compute_flow_magnitude()`.
4. Compare against `flow_threshold` to decide keyframe status.

If fixed-interval mode is active, lines 1245–1257 toggle `self._set_skip_append(True)` for non-keyframes so they do not enlarge the KV cache.

### Prediction Collection and Memory Cleanup

Per-frame outputs are optionally moved to `output_device` to prevent GPU accumulation. Lines 1285–1315 accumulate predictions in Python lists via an internal `_to_out` helper. After the final window, lines 1361–1368 clear the KV cache again and call `torch.cuda.empty_cache()` if CUDA is available.

### Cross-Window Alignment and Stitching

The post-processing block at lines 1443–1505 executes:

- `_pairwise_alignment()` on overlapping keyframes to estimate the transform.
- `_warp_predictions()` to geometrically align neighboring windows.
- `_stitch_windows()` to merge sequences and drop redundant overlap.

The merged dictionary returned at lines 1510–1519 contains `pose_enc`, `depth`, and `world_points`.

## Memory Guarantees for Arbitrary-Length Videos

Because each window resets the KV cache, memory consumption is bounded by the cache capacity of a single window rather than the full video length. Setting `kv_cache_sliding_window=64` ensures that even a 10,000-frame video consumes the same peak VRAM as a 64-frame clip. For extremely long sequences, set `output_device=torch.device('cpu')` to stream per-frame tensors to host memory and keep the GPU footprint minimal.

## Practical Code Examples

### Long Video with Dynamic Keyframes

```python
import torch
from lingbot_map.models.gct_stream_window_v2 import GCTStream

model = GCTStream(
    img_size=518,
    patch_embed='dinov2_vitl14_reg',
    enable_camera=True,
    enable_point=True,
    enable_depth=True,
    sliding_window_size=-1,
    kv_cache_sliding_window=64,
    kv_cache_scale_frames=8,
)
model.eval()
model.to('cuda')

video = torch.randn(5000, 3, 518, 518).to('cuda')

pred = model.inference_windowed(
    images=video,
    window_size=16,
    overlap_keyframes=4,
    keyframe_interval=5,
    flow_threshold=0.8,
    max_non_keyframe_gap=30,
    output_device=torch.device('cpu'),
)

pose_enc = pred["pose_enc"]
depth_maps = pred["depth"]

```

### Fixed-Interval Mode Only

```python
pred = model.inference_windowed(
    images=video,
    window_size=12,
    overlap_size=8,
    keyframe_interval=3,
    flow_threshold=0.0,
    output_device=None,
)

```

### CPU-Only Execution

```python
model.to('cpu')
pred = model.inference_windowed(
    images=video.cpu(),
    window_size=10,
    overlap_keyframes=2,
    keyframe_interval=1,
    output_device=torch.device('cpu'),
)

```

## Key Files in the Windowed Inference Stack

- [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py) — Hosts `GCTStream.inference_windowed()`, window stitching, and flow-based keyframe logic.
- [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py) — Base streaming inference method that the windowed implementation extends.
- [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py) — `AggregatorStream`, the transformer backbone that manages the KV cache.
- [`lingbot_map/heads/camera_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/camera_head.py) — `CameraCausalHead`, which participates in cache skip and rollback operations.
- [`lingbot_map/utils/pose_enc.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/pose_enc.py) — Pose encoding utilities used by the flow magnitude computation.
- [`lingbot_map/utils/geometry.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/geometry.py) — Provides `closed_form_inverse_se3` and helpers for the similarity transform.
- [`lingbot_map/models/gct_base.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_base.py) — Base class defining common forward-pass logic and prediction normalization.

## Summary

- Split the video into overlapping windows and reset the KV cache between each window to bound GPU memory.
- Use bidirectional **scale frames** at the start of every window to preserve spatial consistency.
- Choose between **fixed-interval** and **flow-based** keyframe detection to control cache population.
- Offload per-frame outputs to CPU via `output_device` when processing very long sequences.
- Align and stitch windows with `_pairwise_alignment()`, `_warp_predictions()`, and `_stitch_windows()` to produce a single coherent prediction stream.

## Frequently Asked Questions

### What is the maximum video length supported by windowed inference mode?

There is no hardcoded frame limit. Because memory usage is bounded per window, the method scales to videos with 10,000 frames or more, constrained only by disk space for the final output tensors and available CPU RAM if `output_device` is set to CPU.

### How does the KV cache stay bounded when processing thousands of frames?

`GCTStream.inference_windowed()` calls `self.clean_kv_cache()` before each new window. The cache only retains entries for the current window, capped by `kv_cache_sliding_window` (default 64). This design prevents the linear growth that causes standard streaming inference to fail.

### Should I use `flow_threshold` or `keyframe_interval` for keyframe selection?

Use `flow_threshold > 0` when scene motion is uneven and you want the model to dynamically retain informative frames. Use `flow_threshold = 0.0` with a fixed `keyframe_interval` for deterministic behavior and slightly lower per-frame overhead. Both modes respect `max_non_keyframe_gap` to avoid losing tracking during long static shots.

### Can I run windowed inference on a CPU-only machine?

Yes. Move the model and inputs to CPU, set `output_device=torch.device('cpu')`, and use a small `window_size` so that each forward pass fits in system memory. The windowed design keeps the working set small regardless of total video length.