# Memory Management and Offloading Strategies in LingBot-Map: 6 Techniques for Low-VRAM Streaming 3D Reconstruction

> Explore LingBot-Map's 6 memory management and offloading strategies including KV-caching and CPU offloading to achieve low-VRAM streaming 3D reconstruction on single GPUs.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: best-practices
- Published: 2026-07-06

---

**LingBot-Map combines paged KV-caching, configurable keyframe intervals, windowed inference, CPU offloading, scale-frame reduction, and SDPA fallback to run streaming 3D reconstruction on single GPUs with as little as 4 GiB of VRAM.**

LingBot-Map is an open-source streaming 3D reconstruction system designed to process video sequences containing tens of thousands of frames on consumer hardware. The `Robbyant/lingbot-map` repository implements sophisticated memory management and offloading strategies that bound activation memory and prevent KV-cache overflow during long-form inference.

## Paged KV-Cache Attention with FlashInfer

The core memory reduction comes from **FlashInfer's paged key-value cache**, which stores only a limited number of keyframes (default ≤ 320) rather than the entire sequence. This dramatically shrinks attention memory compared to standard transformer implementations.

According to the source code in [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py) (and its variants [`gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream_window.py) and [`gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream_window_v2.py)), the model creates and accesses this paged cache during streaming inference. When FlashInfer is unavailable, the system automatically falls back to the native PyTorch SDPA implementation, trading speed for compatibility.

## Keyframe Interval Control

The `--keyframe_interval` flag provides explicit control over cache population density. When set to `N`, the system stores only every *N*-th frame as a keyframe in the KV cache.

Non-keyframes continue to generate predictions but are not added to the cache, effectively limiting memory growth for long videos. As documented in the repository's README (lines 1012–1014), this flag works in conjunction with the paged cache to cap maximum memory usage proportional to sequence length divided by the interval value.

## Windowed Inference for Long Sequences

For sequences that would otherwise exceed the RoPE (Rotary Position Embedding) training range, LingBot-Map implements **windowed inference** via the `--mode windowed` flag.

This strategy splits the stream into overlapping windows defined by `--window_size` and `--overlap_keyframes`, resetting the KV cache at each window boundary. The README (lines 995–1002) specifies that this keeps memory bounded regardless of total video length, preventing the positional encoding overflow that typically limits transformer sequence lengths.

## CPU Offloading of Per-Frame Predictions

Heavy per-frame tensor outputs—including depth maps, color values, and camera poses—are moved from GPU to CPU immediately after generation. This is controlled by the `--offload_to_cpu` flag, which is enabled by default.

In [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py) (lines 543–547), the implementation checks this flag and sets `output_device = torch.device("cpu") if args.offload_to_cpu else None`, ensuring tensors leave VRAM before the next frame processes. Disabling this with `--no-offload_to_cpu` keeps everything on GPU for faster access but requires significantly more VRAM.

## Scale-Frame Reduction

The initial "scale" phase creates bidirectional frames that dominate the activation peak. The `--num_scale_frames` parameter (default 8) controls how many of these frames are generated.

Reducing this value lowers the initial memory spike, freeing GPU resources for the subsequent streaming phase where the paged cache takes over. This parameter is documented alongside other memory flags in the README (lines 1012–1014).

## Optional SDPA Fallback

When FlashInfer is not installed, the system automatically switches to the slower but more memory-efficient PyTorch SDPA (Scaled Dot-Product Attention) implementation. Users can explicitly force this fallback using the `--use_sdpa` flag.

As noted in the README (lines 1002–1005), this fallback ensures the model runs on hardware where FlashInfer binaries are unavailable, though with reduced performance compared to the paged cache implementation.

## Practical Configuration Examples

Run the basic demo with default CPU offloading for low-VRAM GPUs:

```bash
python demo.py \
    --model_path /path/to/lingbot-map-long.pt \
    --image_folder example/university \
    --mask_sky

```

Reduce memory by keeping only every second frame as a keyframe:

```bash
python demo.py \
    --model_path /path/to/lingbot-map-long.pt \
    --image_folder example/university \
    --keyframe_interval 2 \
    --offload_to_cpu

```

Process very long videos (> 10,000 frames) using windowed inference:

```bash
python demo.py \
    --model_path /path/to/lingbot-map-long.pt \
    --video_path video.mp4 \
    --mode windowed \
    --window_size 128 \
    --overlap_keyframes 8 \
    --keyframe_interval 13 \
    --offload_to_cpu

```

Force GPU-only execution (requires sufficient VRAM):

```bash
python demo.py \
    --model_path /path/to/lingbot-map-long.pt \
    --image_folder example/university \
    --no-offload_to_cpu

```

## Monitoring Memory Usage

Debug memory consumption using [`gct_profile.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_profile.py), which prints KV-cache size and GPU memory statistics during inference. For systematic evaluation, run [`scripts/benchmark_gct_memory.py`](https://github.com/Robbyant/lingbot-map/blob/main/scripts/benchmark_gct_memory.py) to compare consumption across different keyframe intervals, offloading configurations, and window sizes.

## Summary

- **Paged KV-Cache**: Uses FlashInfer to limit attention memory to ≤ 320 keyframes in `lingbot_map/models/gct_stream*.py`.
- **Keyframe Interval**: The `--keyframe_interval` flag subsamples cache entries to reduce memory linearly with sequence length.
- **Windowed Inference**: Resets caches at window boundaries via `--mode windowed` to handle unlimited sequence lengths without RoPE overflow.
- **CPU Offloading**: Moves per-frame tensors to CPU immediately in [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py) (lines 543–547) via `--offload_to_cpu`.
- **Scale-Frame Control**: Reduces initial activation peak by lowering `--num_scale_frames` from its default of 8.
- **SDPA Fallback**: Automatically switches to PyTorch SDPA when FlashInfer is unavailable, triggered by `--use_sdpa`.

## Frequently Asked Questions

### How does LingBot-Map handle sequences longer than 10,000 frames?

LingBot-Map uses **windowed inference** (`--mode windowed`) combined with keyframe intervals (`--keyframe_interval`). The windowed mode splits the video into overlapping segments defined by `--window_size`, resetting the KV cache at each boundary to prevent overflow. This allows processing of effectively unlimited sequence lengths while keeping memory bounded by the window size rather than the total frame count.

### What is the difference between `--offload_to_cpu` and `--no-offload_to_cpu`?

The `--offload_to_cpu` flag (enabled by default) moves heavy per-frame tensors—including depth, colors, and poses—from GPU to CPU immediately after generation in [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py) (lines 543–547). This reduces VRAM usage but increases CPU-GPU transfer overhead. Using `--no-offload_to_cpu` keeps all tensors on GPU for faster downstream processing but requires significantly more VRAM, typically suitable only for high-end GPUs with 12+ GiB of memory.

### When should I use the SDPA fallback instead of FlashInfer?

Use the `--use_sdpa` flag when FlashInfer is not installed or when running on hardware where FlashInfer kernels are incompatible. While SDPA is slower than the paged KV-cache implementation, it provides a reliable fallback that maintains the same output quality. The system automatically detects FlashInfer availability and falls back to SDPA if necessary, though explicit specification ensures predictable behavior.

### Can I combine multiple memory strategies simultaneously?

Yes, the strategies are designed to operate together. For minimum VRAM usage on long videos, combine `--mode windowed` with `--window_size 128`, `--keyframe_interval 2` or higher, and `--offload_to_cpu`. The [`scripts/benchmark_gct_memory.py`](https://github.com/Robbyant/lingbot-map/blob/main/scripts/benchmark_gct_memory.py) utility can help identify the optimal combination for your specific GPU memory budget and video length requirements.