# How to Optimize Performance for Long-Sequence Inference Exceeding 10,000 Frames in LingBot-Map

> Optimize long-sequence inference over 10000 frames in LingBot-Map. Utilize keyframe KV-cache reduction, windowed inference, FlashInfer, and CPU offloading to slash GPU memory usage.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: performance
- Published: 2026-07-22

---

**To optimize performance for long-sequence inference exceeding 10,000 frames in LingBot-Map, combine keyframe-based KV-cache reduction with `--keyframe_interval`, windowed inference via `--mode windowed`, and the FlashInfer backend, while enabling CPU offloading to keep GPU memory under 6 GB.**

LingBot-Map is a streaming 3-D reconstruction system designed to process video sequences frame-by-frame without batching. When handling videos exceeding 10,000 frames, the linear growth of the key-value (KV) cache in transformer layers quickly exhausts GPU memory and degrades inference speed. The repository provides three complementary optimization mechanisms—keyframe sampling, windowed processing, and paged attention backends—that together enable processing of arbitrarily long sequences on consumer-grade hardware.

## Understanding the KV-Cache Bottleneck in Streaming Inference

During streaming inference, each frame passes through self-attention layers that generate **KV (Key-Value) tensors** cached for subsequent frames. By default, every frame is treated as a keyframe, causing memory consumption to grow linearly with sequence length. For a 10,000-frame video, this often exceeds 12 GB of VRAM, forcing slower CPU offloading or out-of-memory errors. The optimizations below address this by decoupling processing from storage and bounding the cache size.

## Keyframe-Based KV-Cache Reduction

The `--keyframe_interval` parameter reduces cache size by storing only every *N*-th frame’s KV tensors while still processing every frame for pose estimation.

### How Keyframe Interval Works

In [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py) (lines 5045–5068), the `inference_streaming` method evaluates:

```python
is_keyframe = (keyframe_interval <= 1) or ((i - scale_frames) % keyframe_interval == 0)

```

- **Keyframes**: When `is_keyframe` is true, the system calls `_set_skip_append(False)` to retain KV tensors in GPU memory.
- **Non-keyframes**: When false, `_set_skip_append(True)` prevents appending new KV tensors to the cache; the model attends to existing cached keys but discards the new tensors immediately after use.

This reduces the effective cache size to `⌈total_frames / keyframe_interval⌉`, cutting memory usage by a factor of *N*.

### Configuration and Usage

For videos exceeding 20,000 frames on GPUs with less than 12 GB VRAM, set the interval between 3 and 5:

```bash
python demo.py \
    --model_path /path/to/lingbot-map.pt \
    --video_path ~/videos/indoor_walk.mp4 \
    --mode streaming \
    --keyframe_interval 4 \
    --camera_num_iterations 2

```

The console will confirm: `Keyframe streaming enabled: interval=4 (KV cache reduced ~4×)`. This typically yields a 10–15% wall-clock speed improvement due to reduced memory bandwidth pressure.

## Sliding-Window Inference for Unbounded Sequences

The `--mode windowed` option splits long videos into overlapping segments, each with a fresh KV-cache, preventing unbounded memory growth beyond the training horizon of ~320 frames.

### Windowed Pipeline Architecture

Implemented in `inference_windowed` ([`gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream.py), lines 969–1014), the pipeline executes:

1. **Window sizing**: Processes chunks defined by `--window_size` (measured in keyframes, not raw frames).
2. **Overlap computation**: Calculates effective overlap as `max(num_scale_frames, overlap_keyframes * keyframe_interval)` to ensure smooth pose alignment.
3. **Cache reset**: Calls `self.clean_kv_cache()` between windows to free GPU memory.
4. **Global alignment**: Uses `_pairwise_alignment` with scale-ratio and similarity transforms to stitch windows via `_stitch_windows`.

### Parameter Selection Strategy

Balance memory and accuracy using these flags:

- `--window_size 128`: Limits each window to 128 keyframes.
- `--overlap_keyframes 8`: Shares 8 keyframes between adjacent windows for stable trajectory alignment.
- `--keyframe_interval 8`: Expands each window to cover ~1,200 actual frames while keeping KV-cache small.
- `--scale_mode median`: Provides robust scale estimation across overlapping depth maps.

### Practical Example for 25,000 Frames

```bash
python demo.py \
    --model_path /path/to/lingbot-map.pt \
    --video_path ~/videos/indoor_walk.mp4 \
    --mode windowed \
    --window_size 128 \
    --overlap_keyframes 8 \
    --keyframe_interval 10 \
    --camera_num_iterations 2

```

Output indicates the effective coverage: `Actual frames per window ≈ 8 + (128-8)×10 = 1,208`. This configuration processes 25,000 frames in approximately 4 minutes on an RTX 3060 (12 GB) without OOM errors.

## FlashInfer Backend for Paged KV-Cache

FlashInfer replaces PyTorch’s scaled dot-product attention (SDPA) with a **paged KV-cache** implementation, reducing per-frame latency by 2–3× and allowing larger window sizes.

### Installation and Activation

Install the FlashInfer Python package:

```bash
pip install --index-url https://pypi.org/simple flashinfer-python

```

In [`gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream.py) (lines 61–64), the code automatically selects FlashInfer when `use_sdpa=False` (the default), setting `use_flashinfer=not self.use_sdpa`. If FlashInfer is unavailable, the system falls back to SDPA with higher memory usage.

Verify the backend is active by running with `--compile`:

```bash
python demo.py ... --compile

```

Look for the log line: `Using FlashInfer backend for KV cache (paged attention)`.

## Additional Memory and Speed Tweaks

Fine-tune the trade-off between accuracy and resource usage:

- `--offload_to_cpu` (default **on**): Moves per-frame predictions to CPU immediately after generation, freeing GPU memory for active computation.
- `--num_scale_frames 2`: Reduces bidirectional scale frames from the default 8, lowering activation peaks during initialization.
- `--camera_num_iterations 1`: Runs a single refinement pass in the camera head instead of multiple iterations.
- `--output_device cpu`: Stores all outputs directly on CPU RAM, essential for very long sequences when only the final point cloud is required.

## Complete Optimization Configuration

For a 30,000-frame outdoor drive on a single RTX 3060, combine all techniques:

```bash
python demo.py \
    --model_path /path/to/lingbot-map.pt \
    --video_path ~/videos/drive.mp4 \
    --mode windowed \
    --window_size 128 \
    --overlap_keyframes 8 \
    --keyframe_interval 12 \
    --num_scale_frames 2 \
    --camera_num_iterations 1 \
    --output_device cpu \
    --offload_to_cpu

```

**Execution flow:**
1. FlashInfer allocates a paged KV-cache on GPU.
2. Windowing splits 30,000 frames into ~25 windows (~1,200 frames each).
3. Keyframe interval 12 retains only ~8% of KV entries per window.
4. CPU offloading maintains GPU memory below 4 GB throughout inference.
5. Global alignment stitches windows into a consistent trajectory.

This pipeline completes in ~5 minutes with peak GPU usage under 6 GB.

## Summary

- **Keyframe interval** (`--keyframe_interval N`) reduces KV-cache size by 1/N in `inference_streaming` by discarding non-keyframe tensors immediately after use.
- **Windowed inference** (`--mode windowed`) bounds memory usage by processing long sequences in overlapping chunks with fresh caches, implemented in `inference_windowed`.
- **FlashInfer backend** provides paged attention support, cutting latency by 2–3× compared to SDPA.
- **CPU offloading** (`--offload_to_cpu`, `--output_device cpu`) moves intermediate results out of GPU memory, enabling processing of unlimited sequence lengths given sufficient system RAM.

## Frequently Asked Questions

### What is the maximum sequence length LingBot-Map can process?

With windowed inference and CPU offloading enabled, LingBot-Map can process sequences of arbitrary length limited only by available disk space for storing intermediate results. The windowing mechanism in [`gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream.py) resets the KV-cache every `--window_size` keyframes, preventing the memory accumulation that would otherwise cap sequence length at a few thousand frames.

### How much GPU memory is required for 10,000+ frame sequences?

Without optimization, 10,000 frames typically require over 12 GB VRAM and often trigger out-of-memory errors. Using `--keyframe_interval 10`, `--mode windowed`, and `--offload_to_cpu`, you can process 30,000+ frames on a 12 GB GPU with peak usage under 6 GB, or on an 8 GB GPU with more aggressive keyframe sampling (e.g., interval 15-20).

### Does keyframe interval affect reconstruction accuracy?

Non-keyframes are still fully processed for pose estimation and depth prediction; only their KV tensors are discarded after use. Because the model attends to the existing cache during non-keyframe processing, and because overlapping windows in windowed mode provide alignment constraints, accuracy degradation is minimal for intervals up to 10-12. For critical applications, use `--overlap_keyframes 8` or higher to maintain trajectory consistency.

### Can I use windowed inference without installing FlashInfer?

Yes, windowed inference functions independently of the attention backend. However, without FlashInfer, the system falls back to PyTorch SDPA, which consumes approximately 2–3× more memory per window and runs significantly slower. For sequences exceeding 10,000 frames, FlashInfer is strongly recommended to maintain real-time performance.