# Windowed Inference Parameters in LingBot-Map: Complete API Reference

> Explore windowed inference parameters in LingBot-Map including window_size, overlap_keyframes, and more. Optimize memory-efficient video processing with this API reference.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: api-reference
- Published: 2026-07-03

---

**The `inference_windowed` method in LingBot-Map accepts ten configurable parameters—including `window_size`, `overlap_keyframes`, `flow_threshold`, and `num_scale_frames`—that enable memory-efficient processing of arbitrarily long video sequences through sliding window attention and intelligent KV-cache management.**

LingBot-Map is an open-source video understanding framework that implements streaming geometry estimation for long-form content. For processing extended video sequences without exhausting GPU memory, the library provides a specialized windowed inference mode via the `GCTStream` class, specifically implemented in [`lingbot_map/models/gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py).

## Core Parameters for Windowed Inference

The `inference_windowed` method defined at lines 590–714 in [`lingbot_map/models/gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py) exposes the following parameters:

### Input and Output Configuration

- **`images`** (`torch.Tensor`): The input video batch. Accepts shapes `[S, 3, H, W]` for single batch or `[B, S, 3, H, W]` for multiple batches. Pixel values must be normalized to the range `[0, 1]`.

- **`output_device`** (`torch.device`, optional): Specifies where final predictions are stored. Defaults to `None`, which keeps tensors on the model's device. Setting to `torch.device('cpu')` enables CPU offloading to reduce GPU memory pressure during long sequences.

### Window Configuration

- **`window_size`** (`int`, default `16`): Defines the number of keyframes (frames whose KV-cache is retained) processed per window, including scale frames. Larger values increase temporal context but raise memory consumption linearly.

- **`overlap_size`** (`int`, optional): Specifies the length of overlapping actual frames between consecutive windows. If omitted, defaults to the number of scale frames (`num_scale_frames`). This overlap ensures high-quality predictions at window boundaries by supplying the next window with its scale-phase frames.

- **`overlap_keyframes`** (`int`, optional): Alternative overlap specification expressed in keyframes, preferred when `keyframe_interval > 1`. Takes precedence over `overlap_size`. Internally converted to actual frames as `max(num_scale_frames, overlap_keyframes × keyframe_interval)`.

- **`num_scale_frames`** (`int`, optional): Defaults to `self.num_frame_for_scale`. Defines how many initial frames within each window are processed as scale frames using bidirectional attention, providing a reliable depth-scale reference for subsequent frames.

### Keyframe Management

- **`keyframe_interval`** (`int`, default `1`): Determines the frequency of Phase-2 frames becoming keyframes. A value of `1` retains every frame in the KV-cache (original streaming behavior), while larger values reduce cache growth proportionally.

- **`flow_threshold`** (`float`, default `0.0`): Enables dynamic keyframe detection when greater than zero. Calculates the mean optical-flow magnitude in pixels between the current frame and last keyframe; frames exceeding this threshold become keyframes, overriding `keyframe_interval`.

- **`max_non_keyframe_gap`** (`int`, default `30`): In flow-based mode, sets the maximum allowed consecutive non-keyframes before forcing a keyframe to prevent drift.

### Scale Estimation

- **`scale_mode`** (`str`, default `'median'`): Strategy for estimating inter-window scale factors. Currently implements `'median'`, which uses the median depth-ratio across paired keyframes for alignment.

## How Parameters Interact During Execution

Understanding the internal flow of `inference_windowed` helps optimize parameter selection for specific video lengths and memory constraints.

### Window Creation and Scale Phase

The method first determines the effective overlap (`eff_overlap`) based on `overlap_keyframes` (preferred) or `overlap_size`. For each window, the first `num_scale_frames` frames are processed together as `scale_images` with bidirectional attention. Their KV cache is not cleared, enabling cross-scale-frame communication that establishes the depth reference.

### Streaming Phase and Dynamic Keyframes

Remaining frames are processed sequentially. When `keyframe_interval` exceeds `1`, only every N-th frame persists in the KV-cache. If `flow_threshold` is greater than `0.0`, the system temporarily disables cache eviction via `_set_defer_eviction`, computes flow magnitude using `_compute_flow_magnitude`, and conditionally commits frames based on motion. Non-keyframes trigger `_rollback_last_frame` to free cache space immediately.

### Cross-Window Alignment

After each window completes, `_align_and_stitch_windows` aligns adjacent predictions using the overlapping region. The `_stitch_windows` function combines results using depth-ratio scaling (`_depth_ratio_scale`) and pose-based similarity transforms to maintain global consistency.

## Practical Implementation Examples

### Basic Windowed Inference with Fixed Intervals

The following example demonstrates processing a 120-frame video with explicit window sizing and keyframe intervals:

```python
import torch
from lingbot_map.models.gct_stream_window import GCTStream

# Initialize model

model = GCTStream(
    img_size=518,
    patch_size=14,
    embed_dim=1024,
    pretrained_path="path/to/dinov2_weights.pt",
    enable_camera=True,
    enable_depth=True,
    enable_3d_rope=False,
)

model.eval()
model.cuda()

# Prepare dummy video: 120 frames of 3×518×518

B, S, C, H, W = 1, 120, 3, 518, 518
dummy_frames = torch.rand(B, S, C, H, W).cuda()  # Values in [0, 1]

# Run windowed inference

output = model.inference_windowed(
    images=dummy_frames,
    window_size=20,                # 20 keyframes per window

    overlap_keyframes=5,           # 5-keyframe overlap

    num_scale_frames=2,            # First 2 frames are scale frames

    keyframe_interval=4,           # Keep KV every 4th frame

    flow_threshold=0.0,           # Disable flow-based mode

    output_device=torch.device('cpu'),  # Offload to CPU

)

# Extract results

pose_enc = output["pose_enc"]          # [B, S, 9]

depth    = output.get("depth")         # [B, S, H, W, 1]

world_pts = output.get("world_points") # [B, S, H, W, 3]

```

### Flow-Based Dynamic Keyframe Detection

For scenes with variable motion, enable optical-flow-based keyframe selection:

```python
output = model.inference_windowed(
    images=dummy_frames,
    window_size=16,
    overlap_size=4,          # 4-frame overlap

    flow_threshold=1.5,     # Activate flow-based mode (pixel units)

    max_non_keyframe_gap=10,
)

```

In this configuration, the model automatically promotes high-motion frames to keyframes while respecting the overlap constraints, overriding the fixed `keyframe_interval`.

## Summary

- **Windowed inference** in LingBot-Map processes long videos through overlapping windows to maintain bounded memory usage, controlled by the `inference_windowed` method in [`lingbot_map/models/gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py).
- **Memory management** relies on `window_size`, `keyframe_interval`, and `flow_threshold` to determine which frames retain KV-cache entries, while `output_device` enables CPU offloading.
- **Overlap parameters** (`overlap_size` vs `overlap_keyframes`) ensure continuity between windows, with `num_scale_frames` providing bidirectional depth references at window starts.
- **Dynamic keyframes** triggered by `flow_threshold` adapt to motion in the scene, with `max_non_keyframe_gap` preventing drift during static segments.

## Frequently Asked Questions

### What is the difference between overlap_size and overlap_keyframes?

`overlap_size` specifies the actual number of frames to overlap between windows, while `overlap_keyframes` expresses this overlap in terms of keyframe counts. When `keyframe_interval > 1`, use `overlap_keyframes` for more intuitive control; it takes precedence over `overlap_size` and is converted to actual frames via `max(num_scale_frames, overlap_keyframes × keyframe_interval)`.

### How does flow_threshold affect memory usage?

Setting `flow_threshold` greater than `0.0` enables dynamic keyframe detection based on optical flow magnitude. High-motion frames become keyframes and retain KV-cache entries, while static frames are evicted immediately via `_rollback_last_frame`. This can reduce memory usage in static scenes but may increase it during high-motion sequences compared to fixed `keyframe_interval` sampling.

### What happens if I set keyframe_interval to 1?

A `keyframe_interval` of `1` retains every frame in the KV-cache, matching the original streaming behavior of the base `GCTStream` class. This provides maximum temporal context but linearly increases memory consumption with sequence length, negating the memory-bounded benefits of windowed inference for very long videos.

### Where is the windowed inference logic implemented?

The core implementation resides in [`lingbot_map/models/gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py) within the `GCTStream.inference_windowed` method (lines 590–714). Supporting utilities include [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py) for KV-cache backend management, [`lingbot_map/utils/pose_enc.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/pose_enc.py) for pose encoding utilities, and [`lingbot_map/utils/geometry.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/geometry.py) for geometric transformations used in cross-window alignment.