How to Use torch.compile for Faster Inference Performance in LingBot-Map

LingBot-Map accelerates its GCT streaming inference pipeline by compiling hot sub-modules with torch.compile in reduce-overhead mode and warming them up with CUDA-graph capture.

LingBot-Map is an open-source streaming vision model that leverages PyTorch 2.x compilation features to eliminate inference latency. The repository implements a compact opt-in workflow in demo.py and supporting scripts that targets fixed-shape sub-modules of the GCT architecture, delivering significant speed-ups without requiring modifications to model source code.

Understanding the Compilation Strategy

Targeting Hot Sub-Modules

The compile_model helper function in demo.py and scripts/benchmark_gct_memory.py selectively compiles the compute-intensive blocks of the GCTStream aggregator. Rather than compiling the entire model—which would struggle with dynamic KV-cache management—the codebase targets specific static-shape components:

• Frame blocks: agg.frame_blocks[i] — the per-frame processing layers
• Patch embedding blocks: agg.patch_embed.blocks[i] — the initial visual feature extraction
• Global block attention: block.attn_pre and block.attn.proj — the pre-attention and projection layers
• Feed-forward residuals: block.ffn_residual — the FFN skip connections

Each sub-module is wrapped with torch.compile(block, mode="reduce-overhead"), replacing the original reference in the model hierarchy.

Why Reduce-Overhead Mode?

LingBot-Map fixes mode="reduce-overhead" because the GCT model contains many small, dynamic control-flow steps for KV-cache management and per-frame skips. This compilation mode minimizes JIT compilation costs while still fusing heavy linear and attention kernels, making it ideal for streaming workloads where latency consistency matters more than raw throughput on massive batches.

Enabling torch.compile in LingBot-Map

Command-Line Activation

All entry-point scripts accept a --compile argument that triggers the compilation workflow before streaming begins:

python demo.py \
    --image_folder path/to/images \
    --backend flashinfer \
    --dtype auto \
    --compile

The same flag works in the benchmarking utilities:

python scripts/benchmark_gct_memory.py \
    --height 384 --width 518 \
    --frame-counts 64 128 256 \
    --compile

Programmatic Implementation

For custom inference pipelines, replicate the compilation logic found in demo.py:

import torch
from lingbot_map.models.gct_stream import GCTStream

# Initialize model

model = GCTStream(...)
model.eval().cuda()

agg = model.aggregator

# Compile frame blocks

for i, blk in enumerate(agg.frame_blocks):
    agg.frame_blocks[i] = torch.compile(blk, mode="reduce-overhead")

# Compile patch embedding blocks

for i, blk in enumerate(agg.patch_embed.blocks):
    agg.patch_embed.blocks[i] = torch.compile(blk, mode="reduce-overhead")

# Compile global block components

for blk in agg.global_blocks:
    if hasattr(blk, "attn_pre"):
        blk.attn_pre = torch.compile(blk.attn_pre, mode="reduce-overhead")
    if hasattr(blk, "ffn_residual"):
        blk.ffn_residual = torch.compile(blk.ffn_residual, mode="reduce-overhead")
    blk.attn.proj = torch.compile(blk.attn.proj, mode="reduce-overhead")

Note: The scripts automatically configure PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to prevent out-of-memory errors during the compilation phase.

CUDA Graph Warmup Procedure

After compilation, LingBot-Map executes a warmup routine (_warm_streaming in demo.py) to eliminate first-frame overhead. This process calls torch.compiler.cudagraph_mark_step_begin() before executing dummy forwards, recording a CUDA graph that matches the exact tensor shapes of real inference:

def warmup(model, images, dtype):
    torch.compiler.cudagraph_mark_step_begin()
    with torch.no_grad(), torch.amp.autocast("cuda", dtype=dtype):
        model.forward(images[:1])  # Capture graph with representative shapes

warmup(model, synthetic_images, torch.float16)

Once warmed, the streaming loop in inference_streaming runs entirely under the compiled graph, maintaining low latency across variable-length sequences.

Benchmarking Compilation Performance

The gct_profile.py script demonstrates the compilation workflow and reports speed-up statistics when --compile is passed. For comprehensive memory and latency analysis, use scripts/benchmark_gct_memory.py, which prints both peak memory usage and runtime for each frame count, allowing direct comparison between eager and compiled modes.

Summary

• Add --compile to demo.py, benchmark_gct_memory.py, or gct_profile.py to enable the optimized path
• CUDA required: torch.compile only functions on GPU devices; the repository handles memory configuration automatically
• Automatic warmup: The streaming warmup runs automatically when using the entry-point scripts with the compile flag
• Full compatibility: Works safely alongside --backend flashinfer and --dtype auto without source code changes

Frequently Asked Questions

Can I use torch.compile with the FlashInfer backend?

Yes. According to the source code in demo.py and gct_profile.py, the --compile flag is fully compatible with --backend flashinfer. The compilation occurs after model initialization but before the backend-specific streaming loops begin, ensuring the fused kernels operate correctly with FlashInfer's optimized attention implementation.

Why is warmup necessary for streaming inference?

Warmup captures a CUDA graph that matches the exact memory layout and tensor shapes of production inference. Without this step, the first few frames would trigger lazy compilation and graph capture, causing latency spikes. The _warm_streaming function in demo.py executes dummy forwards while calling torch.compiler.cudagraph_mark_step_begin(), ensuring subsequent frames hit the pre-recorded graph immediately.

What should I do if I encounter out-of-memory errors during compilation?

The entry scripts automatically set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to mitigate memory fragmentation during the compilation phase. If you still encounter OOM errors, reduce the batch size or frame dimensions using --height and --width flags in the benchmark scripts, as compilation temporarily requires additional memory for kernel generation.

Is torch.compile supported on CPU devices?

No. The LingBot-Map compilation workflow is designed exclusively for CUDA devices. The compile_model function and associated warmup routines rely on torch.compiler.cudagraph_mark_step_begin() and CUDA-graph capture, which are not available on CPU. Running with --compile on a CPU-only machine will raise a runtime error.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}