# How Windowed Inference Handles Long Sequences in LingBot-Map

> Discover how windowed inference in LingBot-Map tackles long sequences using overlapping windows, KV caches, and bidirectional scale-phase processing for efficient video analysis.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: deep-dive
- Published: 2026-07-06

---

**LingBot-Map processes arbitrarily long video sequences by dividing them into overlapping windows with bounded KV caches, running a bidirectional scale-phase followed by causal streaming, and stitching results using geometric alignment of overlapping keyframes.**

LingBot-Map implements a sliding-window inference pipeline that keeps GPU memory constant regardless of input length. This approach, implemented in the `GCTStream` class within the [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map) repository, enables high-quality pose and depth estimation on videos containing millions of frames without running out of memory.

## Dividing Sequences into Overlapping Windows

The windowed inference algorithm begins in the `inference_windowed` method (lines ~1200-1260 of [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py)) by partitioning the video stream into manageable chunks.

Each window contains a fixed number of **keyframes** (stored in the KV cache) plus **scale frames** that are processed bidirectionally. The `window_size` parameter defaults to 16 keyframes, but the actual number of frames per window follows this formula:

```python
actual_frames = scale_frames + (window_size - scale_frames) × keyframe_interval

```

Consecutive windows overlap by at least the number of scale frames (`overlap_size` defaults to `num_scale_frames`). You can also specify overlap in keyframes using `overlap_keyframes`, which the method internally converts to actual-frame overlap using the keyframe interval.

## KV Cache Management and Memory Bounding

Before processing each window, the model calls `clean_kv_cache()` to clear both the aggregator's and camera head's key-value caches. This guarantees that memory usage never exceeds the size required for a single window, regardless of total video length.

The `AggregatorStream` class in [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py) implements the underlying cache logic (supporting FlashInfer or SDPA backends) and provides the eviction, rollback, and defer-eviction mechanisms used during streaming.

## Two-Phase Processing Within Windows

### Scale-Phase (Phase 1)

The first `num_frame_per_block` frames (equal to `scale_frames`) run together as a batch. Because these frames share a scale token, they attend bidirectionally rather than causally. This yields a robust estimate of the scene scale that improves downstream alignment between windows.

### Streaming Phase (Phase 2)

The remaining frames in the window are processed one-by-one in causal mode:

- **Keyframe mode** (`keyframe_interval > 1`): Only every Nth frame becomes a keyframe and retains its KV cache entry. Non-keyframes are processed with `skip_append=True` so their KV is discarded immediately after the forward pass (see `_set_skip_append`).

- **Flow-based mode** (`flow_threshold > 0`): The model computes optical flow magnitude between the current frame and the latest keyframe using `_compute_flow_magnitude`. If the flow exceeds the threshold, the frame becomes a keyframe; otherwise, the model calls `_rollback_last_frame` and processes it with deferred eviction via `_set_defer_eviction`.

The causal camera head in [`lingbot_map/heads/camera_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/camera_head.py) respects these KV-cache flags during per-frame pose refinement.

## Global Alignment and Result Stitching

After all windows are evaluated, the pipeline reconciles them into a globally consistent reconstruction:

1. **Pairwise alignment**: `_pairwise_alignment` estimates similarity transforms (scale *s*, rotation *R*, translation *t*) using the overlapping keyframes between adjacent windows.

2. **Warping**: Each window is warped into the coordinate frame of the first window using `_warp_predictions`.

3. **Stitching**: Results are concatenated via `_stitch_windows`, which de-duplicates overlapping frames to produce a seamless output.

The final merged prediction dictionary contains the full-length pose encodings, depth maps, world-point clouds, and the original images with shapes preserved across the entire sequence.

## Device Handling and Long-Sequence Optimization

For extremely long sequences, the `inference_windowed` method accepts an `output_device` parameter (e.g., `torch.device('cpu')`). This offloads per-frame outputs from GPU to CPU memory during processing, preventing out-of-memory errors while keeping the model weights on the accelerator for fast inference.

## Practical Implementation Example

```python
import torch
from lingbot_map.models.gct_stream_window_v2 import GCTStream

# Build the model (use a pretrained DINOv2 checkpoint if you have one)

model = GCTStream(
    img_size=518,
    patch_embed='dinov2_vitl14_reg',
    sliding_window_size=-1,          # full causal by default

    num_frame_for_scale=8,           # 8 scale frames per window

    enable_camera=True,
)

# Dummy video: 500 frames, 3×518×518 RGB, batch size 1

video = torch.rand(1, 500, 3, 518, 518)

# Run windowed inference:

preds = model.inference_windowed(
    images=video,
    window_size=16,          # 16 keyframes per window

    overlap_keyframes=4,     # 4‑keyframe overlap → 8 actual‑frame overlap

    keyframe_interval=2,     # store KV every other frame

    flow_threshold=0.0,      # disable flow‑based mode

    output_device=torch.device('cpu')  # keep results on CPU

)

# `preds` now contains full‑length pose_enc, depth, world_points, etc.

print(preds["pose_enc"].shape)   # → torch.Size([1, 500, 9])

```

## Summary

- **Windowed inference** divides long sequences into overlapping chunks with configurable keyframe and scale-frame counts.
- **Bounded memory** is achieved by clearing the KV cache (`clean_kv_cache()`) before each window processes.
- **Bidirectional scale estimation** in Phase 1 establishes robust scene scale, while **causal streaming** in Phase 2 handles the remaining frames with selective KV retention.
- **Geometric alignment** via `_pairwise_alignment` and `_warp_predictions` stitches windows into a globally consistent coordinate frame.
- **CPU offloading** via `output_device` allows processing of millions of frames without GPU memory exhaustion.

## Frequently Asked Questions

### What is the maximum video length LingBot-Map can process?

There is no fixed maximum length. Because windowed inference keeps GPU memory bounded to the size of a single window (controlled by `window_size` and `num_frame_for_scale`), the system can theoretically process millions of frames, limited only by available CPU RAM for storing results when `output_device` is set to CPU.

### How does the keyframe interval affect reconstruction quality?

A smaller `keyframe_interval` stores more frames in the KV cache, improving temporal coherence and accuracy but increasing memory usage per window. A larger interval reduces memory but may miss rapid motion. The flow-based mode (`flow_threshold > 0`) provides an adaptive alternative that inserts keyframes dynamically based on optical flow magnitude computed in `_compute_flow_magnitude`.

### What is the difference between scale frames and keyframes?

**Scale frames** (handled by `num_frame_for_scale`) are the first frames in each window that run bidirectionally to establish scene scale. **Keyframes** (controlled by `window_size` and `keyframe_interval`) are retained in the KV cache throughout the window to maintain temporal context; non-keyframes are processed with `skip_append=True` and discarded immediately to save memory.

### Can windowed inference run on CPU-only machines?

Yes, by setting `output_device=torch.device('cpu')` and ensuring the model itself is on CPU, the pipeline can run without a GPU. However, inference will be significantly slower as the `GCTStream` architecture relies on accelerated attention backends (FlashInfer or SDPA) for efficient processing of the scale and streaming phases.