# How to Optimize Keyframe Intervals for Long Sequence Reconstruction in LingBot-Map

> Optimize keyframe intervals in LingBot-Map for long sequence reconstruction. Tune fidelity and GPU memory usage with keyframe_interval, flow_threshold, and max_non_keyframe_gap parameters.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: performance
- Published: 2026-07-27

---

**LingBot-Map controls memory growth by storing full KV caches only for keyframes, letting you tune the trade-off between reconstruction fidelity and GPU memory usage via the `keyframe_interval`, `flow_threshold`, and `max_non_keyframe_gap` parameters.**

Processing thousands of frames in dense 3D reconstruction typically exhausts GPU memory, but the [LingBot-Map](https://github.com/Robbyant/lingbot-map) repository solves this through a sliding-window streaming architecture where only designated keyframes retain full attention caches. By strategically spacing these keyframes, you can scale to sequences exceeding 10,000 frames while preserving camera pose and depth estimation accuracy. This guide examines the specific implementation details in the source code and provides a practical tuning workflow for long-sequence deployment.

## Understanding Keyframe-Based Memory Management

LingBot-Map implements a **sliding-window stream** where frames are processed sequentially, but only a subset are stored as *keyframes*. According to the implementation in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py), keyframes hold the complete KV (key-value) cache, while non-keyframes attend to the existing cache and then discard their own KV tensors. This design keeps memory growth roughly proportional to the number of keyframes rather than the total frame count, enabling processing of arbitrarily long sequences on limited GPU memory.

The critical decision logic resides in the streaming loop, where the boolean flag `is_keyframe` determines whether a frame's KV cache persists:

```python
is_keyframe = (keyframe_interval <= 1) or ((i - scale_frames) % keyframe_interval == 0)

```

When `is_keyframe` is `False`, the model computes cross-attention using the existing cache but frees the new KV tensors immediately after the forward pass, preventing the linear memory accumulation that typically limits sequence length.

## Keyframe Selection Modes

LingBot-Map offers two complementary mechanisms for selecting which frames become keyframes: fixed-interval scheduling and optical-flow-based adaptive selection.

### Fixed-Interval Mode

In **fixed-interval mode** (`keyframe_interval > 1`), every *N*-th frame after the initial scaling frames becomes a keyframe. This mode is controlled by the `keyframe_interval` parameter in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py). Setting `keyframe_interval = 4` reduces KV memory usage to approximately 25% of a full-cache configuration, making it the baseline for long-sequence reconstruction.

### Flow-Based Adaptive Selection

When `flow_threshold > 0`, the model enables **flow-based mode**, which can override the fixed interval based on scene dynamics. The method `_compute_flow_magnitude` calculates the optical flow magnitude between the current frame and the last keyframe. If the motion exceeds the threshold (measured in pixels), a new keyframe is forced regardless of the fixed interval:

- **Low threshold (≈0.5 px)**: Captures subtle camera movements, preserving accuracy during slow pans
- **High threshold (>2.0 px)**: Only triggers on dramatic scene changes, maximizing memory savings

This adaptive approach ensures that rapid camera movements or dynamic scenes do not suffer from accumulated drift between widely spaced keyframes.

### Maximum Gap Safeguards

To prevent unbounded drift during static scenes, the parameter `max_non_keyframe_gap` (default value 30) caps the number of consecutive non-keyframes. Even if optical flow remains below the threshold, the model forces a keyframe after every 30 frames, guaranteeing periodic anchor points for camera pose and depth alignment.

## Critical Parameters for Long Sequences

When optimizing for sequences exceeding 10,000 frames, three parameters control the memory-quality trade-off:

**`keyframe_interval`** — Determines the base density of keyframes. Larger values reduce the number of full-resolution predictions and KV cache entries, cutting memory usage by a factor of roughly `1 / keyframe_interval`. However, excessively large intervals degrade fine-grained detail in later frames due to accumulated attention drift.

**`flow_threshold`** — Enables the model to dynamically increase keyframe density during fast motion while maintaining sparse sampling during static periods. This preserves reconstruction detail without requiring a universally small interval that would waste memory on redundant frames.

**`max_non_keyframe_gap`** — Provides a hard upper bound on cache drift by guaranteeing a keyframe appears at least every *N* frames. This prevents pose estimation errors from compounding during long static shots where flow-based detection might otherwise skip keyframes indefinitely.

## Recommended Configuration for 10,000+ Frame Sequences

For very long sequences in production environments, apply this three-step tuning strategy implemented in [`scripts/benchmark_gct_memory.py`](https://github.com/Robbyant/lingbot-map/blob/main/scripts/benchmark_gct_memory.py):

1. **Set a modest base interval** — Start with `keyframe_interval = 4`. This reduces KV memory to approximately 25% of the unoptimized baseline while retaining sufficient anchor frames for reliable depth-scale alignment across the sequence.

2. **Enable flow-based selection** — Configure `flow_threshold = 0.5` to automatically trigger additional keyframes during rapid camera movements. This ensures that motion-heavy segments maintain reconstruction quality without forcing a memory-intensive low interval across the entire sequence.

3. **Enforce maximum gaps** — Set `max_non_keyframe_gap = 30` to ensure that even during completely static scenes, a keyframe appears at least every 30 frames. This prevents unbounded drift of camera pose estimates while capping worst-case memory growth.

You can experiment with these trade-offs using the benchmark script's CLI flag: `--keyframe-interval 4`, which passes the value directly to the model constructor.

## Implementation: Configuring the Model

The following configuration instantiates a `GCTStream` model optimized for 15,000-frame sequences with tuned keyframe parameters:

```python
from lingbot_map.models.gct_stream import GCTStream

# Initialize model with long-sequence keyframe optimization

model = GCTStream(
    img_size=518,
    patch_size=14,
    enable_3d_rope=True,
    max_frame_num=15000,                # Support for very long sequences

    kv_cache_sliding_window=64,
    kv_cache_scale_frames=8,
    kv_cache_cross_frame_special=True,
    kv_cache_include_scale_frames=True,
    use_sdpa=False,                     # Use flashinfer backend for speed

    camera_num_iterations=4,
    # Keyframe interval optimization

    keyframe_interval=4,                # Every 4th frame is a keyframe

    flow_threshold=0.5,                # Trigger on motion > 0.5 px

    max_non_keyframe_gap=30,            # Force keyframe every 30 frames max

)

model.eval().to("cuda")

```

During inference, you can inspect which frames were treated as keyframes by examining the `is_keyframe` flag in the model's prediction dictionary: `pred["is_keyframe"]`. This boolean tensor is useful for debugging drift issues or implementing adaptive strategies that adjust the interval on-the-fly based on reconstruction confidence.

## Summary

- **Keyframe architecture**: LingBot-Map stores full KV caches only for keyframes, reducing memory growth from O(N) to O(N/keyframe_interval).
- **Fixed interval**: Controlled in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py) via `keyframe_interval`, with the logic `(i - scale_frames) % keyframe_interval == 0`.
- **Adaptive selection**: Optical flow magnitude calculated by `_compute_flow_magnitude` can force keyframes when motion exceeds `flow_threshold`.
- **Safety bounds**: `max_non_keyframe_gap` guarantees a keyframe at least every 30 frames to prevent unbounded drift.
- **Long-sequence recipe**: Use `keyframe_interval=4`, `flow_threshold=0.5`, and `max_non_keyframe_gap=30` for 10,000+ frame sequences.
- **Validation**: Monitor `pred["is_keyframe"]` during streaming to verify keyframe distribution matches your scene dynamics.

## Frequently Asked Questions

### How does the keyframe interval affect GPU memory usage?

Memory usage scales inversely with the keyframe interval. Because only keyframes retain their KV caches in the sliding window, setting `keyframe_interval = 4` reduces KV memory to approximately 25% of what would be required to cache every frame. The total memory remains bounded regardless of sequence length, enabling processing of 10,000+ frame videos on consumer GPUs.

### What is the optimal flow threshold for handheld camera sequences?

For typical handheld or drone footage with moderate motion, a threshold of **0.5 pixels** provides the best balance. This captures subtle camera movements and prevents drift during slow pans while avoiding excessive keyframe creation during completely static shots. For high-speed action sequences, increase the threshold to 1.0–2.0 pixels to prevent memory spikes, or rely on the `max_non_keyframe_gap` safeguard.

### Why does LingBot-Map require a maximum non-keyframe gap?

Without `max_non_keyframe_gap`, a completely static scene could theoretically proceed indefinitely without creating new keyframes if the optical flow remains below the threshold. Over hundreds of frames, small numerical errors in pose estimation accumulate into significant drift. The gap parameter forces a keyframe every *N* frames (default 30), resetting the reference point and bounding the maximum drift error regardless of scene content.

### Can I change the keyframe interval during inference on a long sequence?

While the base `keyframe_interval` is set at model initialization, you can implement adaptive logic by monitoring the `pred["is_keyframe"]` output and dynamically adjusting the `flow_threshold` or temporarily forcing keyframes for specific segments. For fully dynamic interval changes, you would need to modify the streaming loop in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py) to accept a callable or external controller rather than the fixed integer parameter.