# How to Optimize the Inference Speed Versus Accuracy Trade‑Off in LingBot‑Map: A Complete Guide

> Optimize LingBot-Map inference speed vs. accuracy. Tune flags like keyframe_interval camera_num_iterations and compile for 8 FPS low latency or maximum geometric accuracy. Complete guide.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: how-to-guide
- Published: 2026-06-29

---

**Tune the `--keyframe_interval`, `--camera_num_iterations`, and `--compile` flags to slide between 8 FPS (low latency) and maximum geometric accuracy in LingBot‑Map's transformer vision pipeline.**

The LingBot‑Map repository implements a transformer‑style vision model (GCT) for real‑time camera pose estimation and depth reconstruction. Optimizing the inference speed versus accuracy trade‑off in LingBot‑Map requires understanding how keyframe sampling, KV‑cache management, and iterative refinement interact within the streaming inference pipeline.

## Key Parameters That Control the Speed‑Accuracy Trade‑Off

### Keyframe Interval and KV‑Cache Efficiency

The `--keyframe_interval` argument in [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py) controls how often the model writes to the KV cache. When set to `N`, only every *N*th frame stores new key‑value tensors; intermediate frames read from the cache but skip the expensive write‑back operation.

- **Speed impact**: Reduces memory traffic and GPU kernel launch overhead, directly increasing throughput.
- **Accuracy impact**: Fewer keyframes means less temporal context for pose estimation, causing minor drift in camera trajectory.

In `GCTStream.inference_streaming` (located in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py)), the `_skip_append` flag is toggled for non‑keyframes to prevent cache growth.

### Camera‑Head Iterations and Refinement

The `--camera_num_iterations` parameter sets `self.num_iterations` inside `CameraCausalHead.__init__` ([`lingbot_map/heads/camera_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/camera_head.py)). Each iteration runs a full transformer block to refine the camera pose estimate.

- **Speed impact**: Linear cost per additional iteration; reducing from 4 to 1 can double effective frame rate.
- **Accuracy impact**: Each extra iteration improves pose accuracy by approximately 0.5 % according to the source implementation.

### Mixed‑Precision and Compute Backends

**Mixed‑precision dtype** selection in [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py) automatically uses BF16 on Ampere+ GPUs, falling back to FP16 on older hardware. This cuts tensor memory bandwidth by half while maintaining stability because prediction heads (`_predict_*` methods) internally cast to FP32 for geometric operations.

**Backend selection** (`--use_sdpa`) switches between FlashInfer (paged KV cache with custom CUDA kernels) and a pure‑Python SDPA fallback. FlashInfer delivers approximately 30 % faster inference for long sequences compared to the dictionary‑based cache in `AggregatorStream.__init__` ([`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py)).

### Sliding‑Window Context Management

The `--sliding_window_size` parameter limits self‑attention to the most recent *W* blocks. In `GCTStream.__init__`, this bounds the KV cache size to ensure linear scaling with window size rather than quadratic growth.

- **Speed impact**: Keeps per‑block compute constant regardless of sequence length.
- **Accuracy impact**: Windows that are too small deprive the model of long‑range temporal context, increasing depth and pose drift.

Additional KV‑cache controls (`--kv_cache_sliding_window`, `--num_scale_frames`) manage how many frames are retained for metric scale estimation in `AggregatorStream`, trading memory overhead for geometric stability.

### Compilation and Offloading

**Torch‑compile** (`--compile`) triggers `compile_model` and `_warm_streaming` in [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py), recording CUDA graphs and compiling hot modules with `reduce‑overhead` mode. This yields approximately 5 FPS boost on 518 × 378 inputs (roughly 30 % speedup) with no accuracy penalty.

**CPU offloading** (`--offload_to_cpu`) moves each frame’s output tensors to host memory immediately, reducing GPU peak memory pressure. This can modestly decrease speed if PCIe bandwidth is constrained, but has no effect on accuracy.

## How the Streaming Pipeline Works

### Model Construction

The `GCTStream` class in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py) constructs two core components:

1. **AggregatorStream** ([`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py)): Handles patch embedding, FlashInfer/SDPA KV‑cache management, and sliding‑window logic.
2. **CameraCausalHead** ([`lingbot_map/heads/camera_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/camera_head.py)): Performs iterative pose refinement and optionally applies **3‑D RoPE** (`--enable_3d_rope`) for temporal consistency.

### Streaming Inference Loop

`GCTStream.inference_streaming` processes scale frames first using bidirectional attention, then enters a frame loop. For each incoming frame:

- Optical flow may trigger dynamic keyframe detection.
- Fixed `keyframe_interval` logic determines whether to append to the KV cache (`_skip_append` logic).
- `_defer_eviction` flags prevent premature cache eviction when temporal context is needed.

### Windowed Inference for Long Sequences

For videos exceeding thousands of frames, `GCTStream.inference_windowed` splits the sequence into overlapping chunks. Each window maintains an independent KV cache, and overlaps are stitched using `_pairwise_alignment` similarity transforms. This bounds memory usage while preserving scale consistency across window boundaries.

## Practical Recipes for Different Goals

### Maximum FPS Configuration (Real‑Time Applications)

Use these settings for highest throughput on RTX 4090 or equivalent hardware:

- `--keyframe_interval 12` (reduces KV writes by 92 %).
- `--camera_num_iterations 1` (single refinement pass).
- `--offload_to_cpu` (frees GPU memory immediately).
- `--compile` (enables CUDA graphs and torch‑compile).

This combination minimizes memory traffic and per‑frame compute overhead.

### Balanced Accuracy Setup

For approximately 2× speedup with less than 1 % pose error degradation:

- `--keyframe_interval 6` (moderate temporal context).
- `--camera_num_iterations 4` (default refinement).
- `--no-offload_to_cpu` (keeps tensors on GPU to avoid PCIe bottleneck).

### Offline High‑Accuracy Mode

For batch processing where latency is irrelevant:

- `--keyframe_interval 1` (store every frame for full temporal context).
- `--camera_num_iterations 8` (maximum iterative refinement).
- `--enable_3d_rope` (default, improves pose continuity).
- `--mode windowed` with large `--window_size` and modest `--overlap_keyframes` for videos exceeding 5,000 frames.

## Code Examples

### Fast Streaming Demo (≈ 8 FPS on RTX 3090)

```bash
python demo.py \
    --model_path checkpoints/gct.pt \
    --image_folder /data/seq/ \
    --mode streaming \
    --keyframe_interval 12 \
    --camera_num_iterations 1 \
    --offload_to_cpu \
    --compile

```

This configuration minimizes KV cache growth by writing only every 12th frame, runs a single refinement pass, and leverages CUDA graph warm‑up for maximum throughput.

### Balanced Streaming (≈ 4 FPS, < 1 % Pose Loss)

```bash
python demo.py \
    --model_path checkpoints/gct.pt \
    --image_folder /data/seq/ \
    --mode streaming \
    --keyframe_interval 6 \
    --camera_num_iterations 4 \
    --no-offload_to_cpu

```

The moderate keyframe interval retains sufficient temporal context while the default iteration count maintains pose accuracy.

### Windowed High‑Accuracy Inference for Long Videos

```bash
python demo.py \
    --model_path checkpoints/gct.pt \
    --video_path movie.mp4 \
    --mode windowed \
    --window_size 64 \
    --overlap_keyframes 4 \
    --keyframe_interval 1 \
    --camera_num_iterations 4

```

The window size controls how many keyframes each chunk contains; the overlap ensures scale tokens are shared across windows, enabling accurate cross‑window alignment via `_pairwise_alignment` in `GCTStream`.

### Switching to the SDPA Backend

On machines without FlashInfer support:

```bash
python demo.py \
    --model_path checkpoints/gct.pt \
    --image_folder /data/seq/ \
    --mode streaming \
    --use_sdpa \
    --keyframe_interval 6

```

The SDPA path uses a Python dictionary cache instead of FlashInfer’s paged memory manager, respecting all KV‑cache controls but with reduced memory efficiency.

## Summary

- **Keyframe interval** (`--keyframe_interval`) is the primary lever for trading temporal context for speed; values above 1 skip KV writes on non‑keyframe frames.
- **Camera iterations** (`--camera_num_iterations`) linearly affect compute cost and pose accuracy; reduce to 1 for maximum FPS, increase to 8 for offline quality.
- **FlashInfer backend** (`--use_sdpa=false`) provides ~30 % speedup over the pure‑Python SDPA implementation through paged KV‑cache management.
- **Torch‑compile** (`--compile`) adds ~5 FPS via CUDA graphs and module compilation without affecting geometric output.
- **Sliding windows** and **offloading** manage memory constraints for long sequences, preventing out‑of‑memory errors at the cost of PCIe transfer overhead or reduced context.

## Frequently Asked Questions

### What is the most important parameter for speeding up LingBot‑Map inference?

The `--keyframe_interval` flag provides the largest single gain. Setting it to 12 reduces KV‑cache writes by 92 %, cutting memory bandwidth and kernel launch overhead significantly. Each keyframe still receives full temporal attention, while intermediate frames reuse cached context.

### How much accuracy do I lose by reducing camera‑head iterations?

Reducing `--camera_num_iterations` from 4 to 1 degrades pose accuracy by approximately 1.5 % while doubling throughput. The effect is linear: each iteration adds roughly 0.5 % accuracy improvement according to the `CameraCausalHead.trunk_fn` implementation in [`lingbot_map/heads/camera_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/camera_head.py).

### Does mixed‑precision training affect depth estimation stability?

No. While the transformer backbone runs in BF16 or FP16, the prediction heads (`_predict_camera` and `_predict_depth` in [`lingbot_map/models/gct_base.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_base.py)) cast outputs to FP32 internally. This ensures geometric stability for pose and depth estimates while preserving the 2× memory bandwidth savings of mixed precision.

### When should I use windowed mode instead of streaming?

Use `--mode windowed` when processing sequences longer than 5,000 frames. Streaming mode accumulates KV cache indefinitely, eventually exhausting GPU memory. Windowed mode resets the cache at fixed intervals while using overlapping keyframes (`--overlap_keyframes`) to maintain scale consistency across chunks via `_pairwise_alignment` in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py).