How to Optimize GPU Memory Usage in LingBot-Map: 5 Proven Techniques

LingBot-Map optimizes GPU memory through a paged key-value cache with FlashInfer, configurable sliding windows, strategic keyframe intervals, mixed-precision inference, and lazy input materialization, enabling streaming over tens of thousands of frames without linear memory growth.

The Robbyant/lingbot-map repository implements a streaming 3D reconstruction system that processes video frames sequentially. By default, linear KV-cache growth would exhaust GPU memory when handling long sequences, but the codebase provides specific architectural controls to keep memory usage constant regardless of frame count. This guide explains how to leverage the paged KV-cache implementation and configuration flags to optimize GPU memory usage in LingBot-Map while maintaining real-time inference speeds.

Understanding the KV-Cache Architecture

LingBot-Map relies on three distinct cache components to manage attention across streaming frames. Understanding these structures is essential before tuning memory parameters.

Aggregator and Camera-Head Caches

The model maintains two primary cache types in lingbot_map/models/gct_stream.py:

• Aggregator KV cache: Stores token-level keys and values for all past frames to enable causal attention across the sequence. Managed by the clean_kv_cache and _set_skip_append methods.
• Camera-head KV cache: Holds pose-related keys and values, allowing the camera-head to attend to previous frames without recomputing pose transformations.

By default, these caches grow linearly with frame count, which quickly exhausts GPU memory during long sequences.

FlashInfer Paged Cache Implementation

The recommended approach uses FlashInfer to implement a paged KV cache that swaps old blocks to CPU memory on demand. This configuration lives in lingbot_map/models/gct_stream_window_v2.py and provides O(1) memory growth for long sequences.

When using the FlashInfer backend (--backend flashinfer), only the most recent frames specified by kv_cache_sliding_window occupy GPU memory. Older blocks reside in CPU memory and swap in as needed, keeping GPU usage roughly constant regardless of total frame count.

Configure Sliding Windows and Scale Frames

The GCTStream constructor accepts specific parameters to bound memory usage. In scripts/benchmark_gct_memory.py, the build_model function demonstrates these settings:

model = GCTStream(
    img_size=args.width,
    patch_size=args.patch_size,
    enable_3d_rope=True,
    max_frame_num=max_frame_num,
    kv_cache_sliding_window=args.sliding_window,      # Size of in-GPU window

    kv_cache_scale_frames=args.num_scale_frames,    # Frames always kept (bidirectional context)

    kv_cache_cross_frame_special=True,                # Keep special tokens from evicted frames

    kv_cache_include_scale_frames=True,
    use_sdpa=(args.backend == "sdpa"),
    camera_num_iterations=args.camera_num_iterations,
)

Key parameters:

• kv_cache_sliding_window: Defines how many recent frames stay in GPU memory. Smaller values reduce memory but limit context window.
• kv_cache_scale_frames: Number of scale frames permanently cached. These contain bidirectional context for drift correction.
• kv_cache_cross_frame_special: Preserves special tokens from evicted frames, improving long-range accuracy without retaining full frame data.

Adjust Keyframe Intervals to Limit Cache Growth

During streaming, non-keyframes do not store their KV representations in the cache. This behavior is implemented in GCTStream._set_skip_append within lingbot_map/models/gct_stream.py.

Increase the keyframe interval to reduce the number of cached frames:

python demo.py --keyframe_interval 8   # Retain KV cache for 1 out of every 8 frames

The benchmark script measures this effect via the keyframe_interval argument and reports peak allocated and reserved memory. Higher intervals significantly reduce memory pressure but may slightly decrease reconstruction accuracy for fast camera motions.

Enable Mixed-Precision Inference

Running inference in bfloat16 (bf16) or float16 (fp16) reduces memory usage by approximately 50% compared to float32. The resolve_dtype helper in scripts/benchmark_gct_memory.py automatically selects the best available dtype, but you can force it via CLI:

python demo.py --backend flashinfer --dtype bf16

The benchmark script wraps the forward pass with torch.amp.autocast (lines 31-35) to ensure operations run in the reduced precision without manual tensor conversion.

Optimize Input Materialization and Cache Cleanup

Lazy Input Materialization

By default, --materialize_inputs is off, meaning only a small pool of source images lives on the GPU at any time. Each frame moves to the device just before processing, keeping per-frame activation memory minimal for long sequences:

# Default behavior (recommended for memory optimization)

python demo.py

# Higher memory usage (avoid for long sequences)

python demo.py --materialize_inputs

Explicit Cache Cleanup

When switching video sequences, release all cached KV tensors to prevent memory leakage between runs. The clean_kv_cache() method in lingbot_map/models/gct_stream.py frees GPU memory:

from lingbot_map.models.gct_stream import GCTStream

model = GCTStream(...)

# ... process sequence ...

model.clean_kv_cache()  # Release all KV tensors before next sequence

The benchmark script calls this automatically between test runs.

End-to-End Memory-Optimized Configuration

Combine these settings for the most aggressive memory optimization:

python demo.py \
    --backend flashinfer \
    --dtype bf16 \
    --keyframe_interval 8 \
    --sliding-window 64 \
    --scale-frames 8

This configuration uses the FlashInfer paged cache, limited sliding windows, sparse keyframe storage, and mixed precision to stream thousands of frames on a single GPU.

Benchmarking Memory Usage

Verify your optimizations using the built-in benchmark script:

python scripts/benchmark_gct_memory.py \
    --height 384 --width 518 \
    --patch-size 14 \
    --adjust-to-patch floor \
    --frame-counts 64 128 256 512 1024 2048 4096 10000 \
    --backend flashinfer \
    --dtype bf16 \
    --keyframe-interval 8 \
    --output memory_profile.csv

The resulting CSV reports peak_allocated_gib and peak_reserved_gib, confirming that memory stays under your GPU limit even for 10,000-frame sequences.

Summary

• Use FlashInfer backend: The paged KV cache in gct_stream_window_v2.py prevents linear memory growth by swapping old blocks to CPU.
• Set sliding windows: Configure kv_cache_sliding_window and kv_cache_scale_frames to bound the GPU-resident cache size.
• Increase keyframe intervals: Use --keyframe_interval to reduce the number of frames caching KV tensors, implemented via _set_skip_append.
• Run mixed-precision: Specify --dtype bf16 or fp16 to halve memory usage compared to float32.
• Avoid input materialization: Keep --materialize_inputs disabled (default) and call clean_kv_cache() between sequences to free GPU memory.

Frequently Asked Questions

What is the FlashInfer backend in LingBot-Map?

FlashInfer provides paged KV-cache attention that enables efficient streaming by keeping only a sliding window of recent frames in GPU memory while paging older blocks to CPU. According to the Robbyant/lingbot-map source code, this backend prevents the linear memory growth observed with the SDPA backend, which can suffer from hidden KV growth bugs. Install it via pip install --index-url https://pypi.org/simple flashinfer-python.

How does the sliding window KV cache reduce memory usage?

The kv_cache_sliding_window parameter in GCTStream limits how many recent frames maintain their key-value representations in GPU memory. When combined with FlashInfer, older frames are automatically paged out to CPU and swapped back on demand, resulting in roughly constant memory usage regardless of total frame count. This is implemented in lingbot_map/models/gct_stream_window_v2.py.

When should I use bf16 versus fp16 for inference?

Use bf16 (bfloat16) on Ampere or newer NVIDIA GPUs (A100, RTX 30-series and newer) as it provides better numerical stability than fp16 while offering the same 50% memory reduction. Use fp16 on older hardware where bf16 is unsupported. The resolve_dtype helper in scripts/benchmark_gct_memory.py automatically selects the best available option, but you can force either via --dtype bf16 or --dtype fp16.

How do I clear GPU memory between video sequences?

Call model.clean_kv_cache() after processing each sequence. This method, defined in lingbot_map/models/gct_stream.py, explicitly frees all cached KV tensors from GPU memory. The benchmark script demonstrates this pattern automatically, but custom scripts must invoke it manually to prevent out-of-memory errors when processing multiple videos in a single session.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}