Memory Management and Offloading Strategies in LingBot-Map: 6 Techniques for Low-VRAM Streaming 3D Reconstruction

LingBot-Map combines paged KV-caching, configurable keyframe intervals, windowed inference, CPU offloading, scale-frame reduction, and SDPA fallback to run streaming 3D reconstruction on single GPUs with as little as 4 GiB of VRAM.

LingBot-Map is an open-source streaming 3D reconstruction system designed to process video sequences containing tens of thousands of frames on consumer hardware. The Robbyant/lingbot-map repository implements sophisticated memory management and offloading strategies that bound activation memory and prevent KV-cache overflow during long-form inference.

Paged KV-Cache Attention with FlashInfer

The core memory reduction comes from FlashInfer's paged key-value cache, which stores only a limited number of keyframes (default ≤ 320) rather than the entire sequence. This dramatically shrinks attention memory compared to standard transformer implementations.

According to the source code in lingbot_map/models/gct_stream.py (and its variants gct_stream_window.py and gct_stream_window_v2.py), the model creates and accesses this paged cache during streaming inference. When FlashInfer is unavailable, the system automatically falls back to the native PyTorch SDPA implementation, trading speed for compatibility.

Keyframe Interval Control

The --keyframe_interval flag provides explicit control over cache population density. When set to N, the system stores only every N-th frame as a keyframe in the KV cache.

Non-keyframes continue to generate predictions but are not added to the cache, effectively limiting memory growth for long videos. As documented in the repository's README (lines 1012–1014), this flag works in conjunction with the paged cache to cap maximum memory usage proportional to sequence length divided by the interval value.

Windowed Inference for Long Sequences

For sequences that would otherwise exceed the RoPE (Rotary Position Embedding) training range, LingBot-Map implements windowed inference via the --mode windowed flag.

This strategy splits the stream into overlapping windows defined by --window_size and --overlap_keyframes, resetting the KV cache at each window boundary. The README (lines 995–1002) specifies that this keeps memory bounded regardless of total video length, preventing the positional encoding overflow that typically limits transformer sequence lengths.

CPU Offloading of Per-Frame Predictions

Heavy per-frame tensor outputs—including depth maps, color values, and camera poses—are moved from GPU to CPU immediately after generation. This is controlled by the --offload_to_cpu flag, which is enabled by default.

In demo.py (lines 543–547), the implementation checks this flag and sets output_device = torch.device("cpu") if args.offload_to_cpu else None, ensuring tensors leave VRAM before the next frame processes. Disabling this with --no-offload_to_cpu keeps everything on GPU for faster access but requires significantly more VRAM.

Scale-Frame Reduction

The initial "scale" phase creates bidirectional frames that dominate the activation peak. The --num_scale_frames parameter (default 8) controls how many of these frames are generated.

Reducing this value lowers the initial memory spike, freeing GPU resources for the subsequent streaming phase where the paged cache takes over. This parameter is documented alongside other memory flags in the README (lines 1012–1014).

Optional SDPA Fallback

When FlashInfer is not installed, the system automatically switches to the slower but more memory-efficient PyTorch SDPA (Scaled Dot-Product Attention) implementation. Users can explicitly force this fallback using the --use_sdpa flag.

As noted in the README (lines 1002–1005), this fallback ensures the model runs on hardware where FlashInfer binaries are unavailable, though with reduced performance compared to the paged cache implementation.

Practical Configuration Examples

Run the basic demo with default CPU offloading for low-VRAM GPUs:

python demo.py \
    --model_path /path/to/lingbot-map-long.pt \
    --image_folder example/university \
    --mask_sky

Reduce memory by keeping only every second frame as a keyframe:

python demo.py \
    --model_path /path/to/lingbot-map-long.pt \
    --image_folder example/university \
    --keyframe_interval 2 \
    --offload_to_cpu

Process very long videos (> 10,000 frames) using windowed inference:

python demo.py \
    --model_path /path/to/lingbot-map-long.pt \
    --video_path video.mp4 \
    --mode windowed \
    --window_size 128 \
    --overlap_keyframes 8 \
    --keyframe_interval 13 \
    --offload_to_cpu

Force GPU-only execution (requires sufficient VRAM):

python demo.py \
    --model_path /path/to/lingbot-map-long.pt \
    --image_folder example/university \
    --no-offload_to_cpu

Monitoring Memory Usage

Debug memory consumption using gct_profile.py, which prints KV-cache size and GPU memory statistics during inference. For systematic evaluation, run scripts/benchmark_gct_memory.py to compare consumption across different keyframe intervals, offloading configurations, and window sizes.

Summary

• Paged KV-Cache: Uses FlashInfer to limit attention memory to ≤ 320 keyframes in lingbot_map/models/gct_stream*.py.
• Keyframe Interval: The --keyframe_interval flag subsamples cache entries to reduce memory linearly with sequence length.
• Windowed Inference: Resets caches at window boundaries via --mode windowed to handle unlimited sequence lengths without RoPE overflow.
• CPU Offloading: Moves per-frame tensors to CPU immediately in demo.py (lines 543–547) via --offload_to_cpu.
• Scale-Frame Control: Reduces initial activation peak by lowering --num_scale_frames from its default of 8.
• SDPA Fallback: Automatically switches to PyTorch SDPA when FlashInfer is unavailable, triggered by --use_sdpa.

Frequently Asked Questions

How does LingBot-Map handle sequences longer than 10,000 frames?

LingBot-Map uses windowed inference (--mode windowed) combined with keyframe intervals (--keyframe_interval). The windowed mode splits the video into overlapping segments defined by --window_size, resetting the KV cache at each boundary to prevent overflow. This allows processing of effectively unlimited sequence lengths while keeping memory bounded by the window size rather than the total frame count.

What is the difference between --offload_to_cpu and --no-offload_to_cpu?

The --offload_to_cpu flag (enabled by default) moves heavy per-frame tensors—including depth, colors, and poses—from GPU to CPU immediately after generation in demo.py (lines 543–547). This reduces VRAM usage but increases CPU-GPU transfer overhead. Using --no-offload_to_cpu keeps all tensors on GPU for faster downstream processing but requires significantly more VRAM, typically suitable only for high-end GPUs with 12+ GiB of memory.

When should I use the SDPA fallback instead of FlashInfer?

Use the --use_sdpa flag when FlashInfer is not installed or when running on hardware where FlashInfer kernels are incompatible. While SDPA is slower than the paged KV-cache implementation, it provides a reliable fallback that maintains the same output quality. The system automatically detects FlashInfer availability and falls back to SDPA if necessary, though explicit specification ensures predictable behavior.

Can I combine multiple memory strategies simultaneously?

Yes, the strategies are designed to operate together. For minimum VRAM usage on long videos, combine --mode windowed with --window_size 128, --keyframe_interval 2 or higher, and --offload_to_cpu. The scripts/benchmark_gct_memory.py utility can help identify the optimal combination for your specific GPU memory budget and video length requirements.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}