# How to Optimize GPU Memory Usage in LingBot-Map: 5 Proven Techniques

> Optimize GPU memory in LingBotMap with 5 techniques: paged key-value cache FlashInfer, sliding windows, keyframe intervals, mixed-precision inference, and lazy input. Stream thousands of frames without memory growth.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: performance
- Published: 2026-07-03

---

**LingBot-Map optimizes GPU memory through a paged key-value cache with FlashInfer, configurable sliding windows, strategic keyframe intervals, mixed-precision inference, and lazy input materialization, enabling streaming over tens of thousands of frames without linear memory growth.**

The `Robbyant/lingbot-map` repository implements a streaming 3D reconstruction system that processes video frames sequentially. By default, linear KV-cache growth would exhaust GPU memory when handling long sequences, but the codebase provides specific architectural controls to keep memory usage constant regardless of frame count. This guide explains how to leverage the paged KV-cache implementation and configuration flags to optimize GPU memory usage in LingBot-Map while maintaining real-time inference speeds.

## Understanding the KV-Cache Architecture

LingBot-Map relies on three distinct cache components to manage attention across streaming frames. Understanding these structures is essential before tuning memory parameters.

### Aggregator and Camera-Head Caches

The model maintains two primary cache types in [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py):

- **Aggregator KV cache**: Stores token-level keys and values for all past frames to enable causal attention across the sequence. Managed by the `clean_kv_cache` and `_set_skip_append` methods.
- **Camera-head KV cache**: Holds pose-related keys and values, allowing the camera-head to attend to previous frames without recomputing pose transformations.

By default, these caches grow linearly with frame count, which quickly exhausts GPU memory during long sequences.

### FlashInfer Paged Cache Implementation

The recommended approach uses **FlashInfer** to implement a paged KV cache that swaps old blocks to CPU memory on demand. This configuration lives in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py) and provides *O(1)* memory growth for long sequences.

When using the FlashInfer backend (`--backend flashinfer`), only the most recent frames specified by `kv_cache_sliding_window` occupy GPU memory. Older blocks reside in CPU memory and swap in as needed, keeping GPU usage roughly constant regardless of total frame count.

## Configure Sliding Windows and Scale Frames

The `GCTStream` constructor accepts specific parameters to bound memory usage. In [`scripts/benchmark_gct_memory.py`](https://github.com/Robbyant/lingbot-map/blob/main/scripts/benchmark_gct_memory.py), the `build_model` function demonstrates these settings:

```python
model = GCTStream(
    img_size=args.width,
    patch_size=args.patch_size,
    enable_3d_rope=True,
    max_frame_num=max_frame_num,
    kv_cache_sliding_window=args.sliding_window,      # Size of in-GPU window

    kv_cache_scale_frames=args.num_scale_frames,    # Frames always kept (bidirectional context)

    kv_cache_cross_frame_special=True,                # Keep special tokens from evicted frames

    kv_cache_include_scale_frames=True,
    use_sdpa=(args.backend == "sdpa"),
    camera_num_iterations=args.camera_num_iterations,
)

```

**Key parameters:**
- **`kv_cache_sliding_window`**: Defines how many recent frames stay in GPU memory. Smaller values reduce memory but limit context window.
- **`kv_cache_scale_frames`**: Number of scale frames permanently cached. These contain bidirectional context for drift correction.
- **`kv_cache_cross_frame_special`**: Preserves special tokens from evicted frames, improving long-range accuracy without retaining full frame data.

## Adjust Keyframe Intervals to Limit Cache Growth

During streaming, non-keyframes do not store their KV representations in the cache. This behavior is implemented in `GCTStream._set_skip_append` within [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py).

Increase the keyframe interval to reduce the number of cached frames:

```bash
python demo.py --keyframe_interval 8   # Retain KV cache for 1 out of every 8 frames

```

The benchmark script measures this effect via the `keyframe_interval` argument and reports peak allocated and reserved memory. Higher intervals significantly reduce memory pressure but may slightly decrease reconstruction accuracy for fast camera motions.

## Enable Mixed-Precision Inference

Running inference in **bfloat16** (`bf16`) or **float16** (`fp16`) reduces memory usage by approximately 50% compared to `float32`. The `resolve_dtype` helper in [`scripts/benchmark_gct_memory.py`](https://github.com/Robbyant/lingbot-map/blob/main/scripts/benchmark_gct_memory.py) automatically selects the best available dtype, but you can force it via CLI:

```bash
python demo.py --backend flashinfer --dtype bf16

```

The benchmark script wraps the forward pass with `torch.amp.autocast` (lines 31-35) to ensure operations run in the reduced precision without manual tensor conversion.

## Optimize Input Materialization and Cache Cleanup

### Lazy Input Materialization

By default, `--materialize_inputs` is **off**, meaning only a small pool of source images lives on the GPU at any time. Each frame moves to the device just before processing, keeping per-frame activation memory minimal for long sequences:

```bash

# Default behavior (recommended for memory optimization)

python demo.py

# Higher memory usage (avoid for long sequences)

python demo.py --materialize_inputs

```

### Explicit Cache Cleanup

When switching video sequences, release all cached KV tensors to prevent memory leakage between runs. The `clean_kv_cache()` method in [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py) frees GPU memory:

```python
from lingbot_map.models.gct_stream import GCTStream

model = GCTStream(...)

# ... process sequence ...

model.clean_kv_cache()  # Release all KV tensors before next sequence

```

The benchmark script calls this automatically between test runs.

## End-to-End Memory-Optimized Configuration

Combine these settings for the most aggressive memory optimization:

```bash
python demo.py \
    --backend flashinfer \
    --dtype bf16 \
    --keyframe_interval 8 \
    --sliding-window 64 \
    --scale-frames 8

```

This configuration uses the FlashInfer paged cache, limited sliding windows, sparse keyframe storage, and mixed precision to stream thousands of frames on a single GPU.

## Benchmarking Memory Usage

Verify your optimizations using the built-in benchmark script:

```bash
python scripts/benchmark_gct_memory.py \
    --height 384 --width 518 \
    --patch-size 14 \
    --adjust-to-patch floor \
    --frame-counts 64 128 256 512 1024 2048 4096 10000 \
    --backend flashinfer \
    --dtype bf16 \
    --keyframe-interval 8 \
    --output memory_profile.csv

```

The resulting CSV reports `peak_allocated_gib` and `peak_reserved_gib`, confirming that memory stays under your GPU limit even for 10,000-frame sequences.

## Summary

- **Use FlashInfer backend**: The paged KV cache in [`gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream_window_v2.py) prevents linear memory growth by swapping old blocks to CPU.
- **Set sliding windows**: Configure `kv_cache_sliding_window` and `kv_cache_scale_frames` to bound the GPU-resident cache size.
- **Increase keyframe intervals**: Use `--keyframe_interval` to reduce the number of frames caching KV tensors, implemented via `_set_skip_append`.
- **Run mixed-precision**: Specify `--dtype bf16` or `fp16` to halve memory usage compared to float32.
- **Avoid input materialization**: Keep `--materialize_inputs` disabled (default) and call `clean_kv_cache()` between sequences to free GPU memory.

## Frequently Asked Questions

### What is the FlashInfer backend in LingBot-Map?

FlashInfer provides paged KV-cache attention that enables efficient streaming by keeping only a sliding window of recent frames in GPU memory while paging older blocks to CPU. According to the `Robbyant/lingbot-map` source code, this backend prevents the linear memory growth observed with the SDPA backend, which can suffer from hidden KV growth bugs. Install it via `pip install --index-url https://pypi.org/simple flashinfer-python`.

### How does the sliding window KV cache reduce memory usage?

The `kv_cache_sliding_window` parameter in `GCTStream` limits how many recent frames maintain their key-value representations in GPU memory. When combined with FlashInfer, older frames are automatically paged out to CPU and swapped back on demand, resulting in roughly constant memory usage regardless of total frame count. This is implemented in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py).

### When should I use bf16 versus fp16 for inference?

Use **bf16** (bfloat16) on Ampere or newer NVIDIA GPUs (A100, RTX 30-series and newer) as it provides better numerical stability than fp16 while offering the same 50% memory reduction. Use **fp16** on older hardware where bf16 is unsupported. The `resolve_dtype` helper in [`scripts/benchmark_gct_memory.py`](https://github.com/Robbyant/lingbot-map/blob/main/scripts/benchmark_gct_memory.py) automatically selects the best available option, but you can force either via `--dtype bf16` or `--dtype fp16`.

### How do I clear GPU memory between video sequences?

Call `model.clean_kv_cache()` after processing each sequence. This method, defined in [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py), explicitly frees all cached KV tensors from GPU memory. The benchmark script demonstrates this pattern automatically, but custom scripts must invoke it manually to prevent out-of-memory errors when processing multiple videos in a single session.