Memory Optimization Strategies for Limited VRAM in Lingbot‑Map: A Complete Guide

Lingbot‑Map reduces VRAM usage by combining mixed‑precision inference, gradient checkpointing, FlashInfer KV‑caching, and selective head disabling, enabling dense 3‑D vision inference on GPUs with as little as 4 GB of memory.

Lingbot‑Map (GCT Streaming) is a dense 3‑D vision model that can quickly exhaust the memory capacity of single‑GPU setups. The Robbyant/lingbot‑map repository provides a suite of orthogonal optimization techniques that together make inference feasible on hardware with strict VRAM constraints. Below are the specific strategies implemented in the codebase, complete with file references and runnable configuration examples.

Mixed‑Precision Autocast

The resolve_dtype helper in scripts/benchmark_gct_memory.py (lines 20‑34) automatically selects the optimal low‑precision format for your hardware. On Ampere‑generation GPUs or newer, it defaults to torch.bfloat16; otherwise, it falls back to torch.float16. The make_autocast context wrapper then runs the forward pass in the chosen precision, cutting activation memory roughly in half.

from scripts.benchmark_gct_memory import resolve_dtype, make_autocast
import torch

dtype = resolve_dtype()  # bfloat16 on Ampere+, else float16

with make_autocast(dtype):
    output = model.forward(frame_tensor)

Gradient Checkpointing

Instead of storing the full activation graph during the forward pass, the model saves only strategic checkpoints and recomputes intermediate tensors during backpropagation. This is controlled by the use_gradient_checkpoint parameter in lingbot_map/models/gct_base.py (line 63), which propagates to all transformer blocks.

The actual checkpointing logic uses torch.utils.checkpoint and appears in:

• lingbot_map/layers/vision_transformer.py (lines 45‑48)
• lingbot_map/aggregator/base.py (lines 103‑108)

model = GCTStream(
    img_size=518,
    patch_size=14,
    use_gradient_checkpoint=True  # Enable checkpointing

).eval().to('cuda')

FlashInfer KV‑Cache

FlashInfer provides a paged key‑value cache that stores only the most recent tokens, drastically reducing memory requirements for long sliding windows. The cache manager is instantiated when use_flashinfer=True (the default path) in lingbot_map/layers/flashinfer_cache.py.

Configuration happens in lingbot_map/aggregator/stream.py (lines 87‑95):

aggregator = StreamAggregator(
    use_sdpa=False,                 # False enables FlashInfer

    kv_cache_sliding_window=64,
    kv_cache_scale_frames=8,
)

Model‑Wide Compilation

The benchmark script applies torch.compile(..., mode="reduce-overhead") to transformer blocks and patch‑embedding modules, yielding faster kernels with tighter intermediate allocations. This is implemented in scripts/benchmark_gct_memory.py (lines 62‑73).

model = torch.compile(model, mode="reduce-overhead")

Explicit Cache Cleanup

Before each benchmark run, the codebase empties the CUDA cache and resets peak‑memory statistics to prevent pollution from previous allocations. The cleanup() function in scripts/benchmark_gct_memory.py (lines 37‑42) handles this:

import gc
import torch

def cleanup():
    gc.collect()
    if torch.cuda.is_available():
        torch.cuda.empty_cache()
        torch.cuda.reset_peak_memory_stats()

CPU‑Side Synthetic Inputs

To prevent the GPU from permanently holding thousands of frames, synthetic source images are created on the CPU and transferred to the GPU only when needed. This pattern appears in scripts/benchmark_gct_memory.py within the make_source_images function (lines 76‑89):

# Create on CPU first

frames = torch.randn(batch_size, num_frames, 3, 518, 518, device="cpu")

# Move to GPU only during inference

frame_tensor = frames.to('cuda', non_blocking=True)

Model Reuse Across Benchmarks

The --keep-model-between-counts flag allows the benchmark script to rebuild the model once and reuse it across different frame‑count tests, avoiding repeated allocation of large model weights. This logic is found in scripts/benchmark_gct_memory.py (lines 101‑109).

# When keep_model_between_counts=True

if model is None:
    model = build_model(config)

# Reuse existing model for next benchmark

Selective Head Disabling

The base constructor in lingbot_map/models/gct_base.py (constructor arguments 48‑53) lets you disable heavy prediction heads (point, depth, or camera) to reduce token streams and associated memory:

model = GCTStream(
    enable_point=False,
    enable_depth=False,
    enable_camera=True,
    # ... other args

)

Sliding‑Window and Scale‑Frame Controls

By limiting kv_cache_sliding_window and kv_cache_scale_frames, you directly control the KV‑cache footprint. These parameters are set during model creation in scripts/benchmark_gct_memory.py (lines 48‑56):

model = GCTStream(
    kv_cache_sliding_window=64,   # Limit cached frames

    kv_cache_scale_frames=8,      # Control scale tokens

    max_frame_num=1024,
)

Summary

• Mixed‑precision inference via resolve_dtype and make_autocast halves activation memory by using bfloat16 or float16.
• Gradient checkpointing recomputes activations during backward passes, enabled by use_gradient_checkpoint in gct_base.py.
• FlashInfer KV‑cache provides paged memory management for long sequences when use_sdpa=False.
• Torch compilation with reduce‑overhead mode optimizes kernel allocations.
• Aggressive cache management via torch.cuda.empty_cache() prevents memory fragmentation between runs.
• CPU‑offloaded data generation keeps synthetic inputs off the GPU until inference time.
• Model reuse across benchmarks avoids reloading weights repeatedly.
• Head pruning and sliding‑window limits reduce the number of active token streams and cache size.

Frequently Asked Questions

How do I enable mixed‑precision training in Lingbot‑Map?

You do not need to manually set the dtype. Import resolve_dtype and make_autocast from scripts/benchmark_gct_memory.py. The helper detects your GPU architecture and returns torch.bfloat16 for Ampere+ or torch.float16 for older cards. Wrap your forward pass in the make_autocast context to automatically cast operations to the lower precision.

What is the minimum VRAM required to run Lingbot‑Map?

By combining all optimization strategies—mixed‑precision, gradient checkpointing, FlashInfer caching, and disabling heavy heads—the model can run on GPUs with as little as 4 GB of VRAM. Performance and maximum frame count will vary based on which optimizations you enable and the specific architecture of your GPU.

Where is gradient checkpointing implemented in the codebase?

Gradient checkpointing is controlled in lingbot_map/models/gct_base.py at line 63 via the use_gradient_checkpoint parameter. The actual torch.utils.checkpoint calls reside in lingbot_map/layers/vision_transformer.py (lines 45‑48) for the ViT transformer and lingbot_map/aggregator/base.py (lines 103‑108) for the token aggregation pipeline.

How does FlashInfer reduce memory compared to standard SDPA?

FlashInfer replaces the standard scaled dot‑product attention (SDPA) backend with a paged KV‑cache that stores only recent tokens rather than the full sequence history. When use_sdpa=False is set in StreamAggregator, the system uses the implementation in lingbot_map/layers/flashinfer_cache.py, which significantly reduces memory for long sliding windows by paging cache entries.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}