LingBot-Map Inference Modes: Streaming vs. Windowed Comparison

Use --mode streaming for low-latency, frame-by-frame processing that maintains a key-value (KV) cache of attention activations, or choose --mode windowed to split long sequences into overlapping windows with bounded GPU memory and global alignment.

The Robbyant/lingbot-map repository implements a generalized cross-transformer architecture for 3D scene reconstruction from image sequences. Understanding these LingBot-Map inference modes is essential for optimizing memory usage, processing speed, and reconstruction quality based on your specific hardware constraints and video length.

How the CLI Selects the Inference Mode

The command-line interface in [demo.py](https://github.com/Robbyant/lingbot-map/blob/main/demo.py) exposes both modes through a single argument. Lines 60-63 define the parser choice:

parser.add_argument(
    "--mode", type=str, default="streaming",
    choices=["streaming", "windowed"],
    help=(
        "streaming: frame‑by‑frame with KV cache; "
        "windowed: overlapping windows for long sequences"
    )
)

The dispatch logic around lines 640-660 forwards the flag to the appropriate method:

if args.mode == "streaming":
    predictions = model.inference_streaming(...)
else:
    predictions = model.inference_windowed(...)

Streaming Mode: Frame-by-Frame with KV Cache

The streaming mode processes sequences incrementally while storing compressed attention maps in a KV cache. This approach minimizes per-frame GPU memory to approximately the size of a single frame plus the growing cache.

Implementation Details

The core implementation resides in [lingbot_map/models/gct_stream.py](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py). The inference_streaming method (lines 49-73) executes three phases:

1. Scale-frame initialization – Processes the first num_scale_frames (default 8) together using bidirectional attention via a special scale token.
2. KV cache cleanup – Calls self.clean_kv_cache() to reset state from previous runs.
3. Per-frame streaming loop – Iterates through remaining frames (lines 447-493), checking keyframe_interval to determine whether to store the frame in the KV cache or skip appending via _set_skip_append(True).

def inference_streaming(
    self,
    images: torch.Tensor,
    num_scale_frames: Optional[int] = None,
    keyframe_interval: int = 1,
    output_device: Optional[torch.device] = None,
) -> Dict[str, torch.Tensor]:
    # Scale-frame phase

    scale_output = self.forward(scale_images, ...)
    # Streaming loop

    for i in range(scale_frames, S):
        is_keyframe = (keyframe_interval <= 1) or ((i - scale_frames) % keyframe_interval == 0)
        if not is_keyframe:
            self._set_skip_append(True)
        frame_output = self.forward(frame_image, ...)
        if not is_keyframe:
            self._set_skip_append(False)
        # Accumulate predictions...

Memory and Performance Characteristics

• GPU memory: Scales linearly with the number of stored keyframes, not the total sequence length.
• Latency: Extremely low per-frame delay since only the current frame resides in active GPU memory.
• Throttling: Increase --keyframe_interval (e.g., to 6) to store only every Nth frame, drastically reducing cache size at the cost of minor accuracy degradation.

Windowed Mode: Overlapping Windows for Long Sequences

The windowed mode splits long videos into overlapping chunks, processes each with a fresh KV cache, then aligns and stitches the results. This is essential for sequences where a single streaming run would exhaust GPU memory.

Window Geometry and Alignment

Implemented in [lingbot_map/models/gct_stream_window.py](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py), the inference_windowed method (lines 59-73) calculates window geometry at lines 613-639:

• window_size: Number of keyframes per window (including scale frames).
• overlap_keyframes: Determines how many frames duplicate between consecutive windows to ensure smooth transitions.

The algorithm processes each window independently (lines 682-735), then performs _pairwise_alignment and _stitch_windows (lines 822-860) to merge predictions into a single continuous trajectory.

When to Use Windowed Inference

• Very long videos (≥500 frames or minutes of footage) where streaming KV cache would exceed GPU memory.
• High-quality global reconstruction requiring drift correction across the entire sequence.
• Bounded memory: Peak GPU usage is capped by window_size rather than growing with video length.

Practical Usage Examples

Streaming Example with Keyframe Interval

For short-to-moderate sequences with reduced memory footprint:

python examples/demo.py \
    --model_path checkpoints/gct.pt \
    --image_folder data/seq1/ \
    --mode streaming \
    --keyframe_interval 6 \
    --offload_to_cpu

This stores only every 6th frame in the KV cache while processing all frames.

Windowed Example for Long Videos

For 2-minute drone footage at 10 FPS:

python examples/demo.py \
    --model_path checkpoints/gct.pt \
    --video_path data/flight.mp4 \
    --fps 10 \
    --mode windowed \
    --window_size 64 \
    --overlap_keyframes 8 \
    --keyframe_interval 2

Here, --window_size 64 limits each chunk to 64 keyframes, while 8 overlapping keyframes ensure smooth alignment between windows.

Direct Python API Usage

You can invoke either mode directly from Python:

import torch
from lingbot_map.models.gct_stream import GCTStream

# Load model

model = GCTStream(
    img_size=518,
    patch_size=14,
    enable_3d_rope=True,
    num_scale_frames=8,
).eval().to("cuda")

# Streaming inference

pose_stream = model.inference_streaming(
    images,
    num_scale_frames=8,
    keyframe_interval=1,
    output_device=torch.device("cpu")
)

# Windowed inference

pose_windowed = model.inference_windowed(
    images,
    window_size=64,
    overlap_keyframes=8,
    keyframe_interval=2,
    output_device=torch.device("cpu")
)

Summary

• Streaming mode (--mode streaming) maintains a persistent KV cache for frame-by-frame processing, offering the lowest latency and linear memory growth suitable for short-to-moderate sequences.
• Windowed mode (--mode windowed) partitions sequences into overlapping windows with independent KV caches, providing bounded GPU memory for long videos at the cost of additional alignment overhead.
• Both modes share the GCTStream architecture and support --keyframe_interval to trade memory for accuracy by skipping KV cache updates on non-critical frames.
• Use --offload_to_cpu in either mode to move predictions to system RAM immediately after computation.

Frequently Asked Questions

How do I choose between streaming and windowed modes in LingBot-Map?

Choose --mode streaming when processing short sequences (under ~500 frames) or when you need minimal latency per frame, such as real-time robot perception. Choose --mode windowed when processing long videos that would exceed GPU memory with a single KV cache, or when you require global alignment across the entire sequence to minimize drift.

What is the KV cache and why does it affect memory usage?

The KV cache (key-value cache) stores compressed attention activations from previous frames to enable causal processing without recomputing past states. In streaming mode, this cache grows with each keyframe, potentially consuming large GPU memory for long videos. Windowed mode resets this cache for each window, bounding memory usage to the window size.

Can I adjust memory usage without switching modes?

Yes. In streaming mode, increase --keyframe_interval to store only every Nth frame in the cache (e.g., --keyframe_interval 6). In windowed mode, reduce --window_size to process smaller chunks. Both approaches reduce GPU memory at the cost of potential accuracy degradation due to fewer stored attention states.

Does windowed mode produce lower quality results than streaming?

Windowed mode typically produces higher quality for very long sequences because the alignment step (_pairwise_alignment) corrects drift between windows. However, it requires an extra stitching computation, making it slightly slower than pure streaming for short sequences where memory is not a constraint.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}