# LingBot-Map Inference Modes: Streaming vs. Windowed Comparison

> LingBot Map inference modes explained: Compare streaming for low-latency processing versus windowed for bounded GPU memory and global alignment. Optimize your KV cache.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: comparison
- Published: 2026-07-08

---

**Use `--mode streaming` for low-latency, frame-by-frame processing that maintains a key-value (KV) cache of attention activations, or choose `--mode windowed` to split long sequences into overlapping windows with bounded GPU memory and global alignment.**

The Robbyant/lingbot-map repository implements a generalized cross-transformer architecture for 3D scene reconstruction from image sequences. Understanding these LingBot-Map inference modes is essential for optimizing memory usage, processing speed, and reconstruction quality based on your specific hardware constraints and video length.

## How the CLI Selects the Inference Mode

The command-line interface in [[`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py)](https://github.com/Robbyant/lingbot-map/blob/main/demo.py) exposes both modes through a single argument. Lines **[60-63](https://github.com/Robbyant/lingbot-map/blob/main/demo.py#L60-L63)** define the parser choice:

```python
parser.add_argument(
    "--mode", type=str, default="streaming",
    choices=["streaming", "windowed"],
    help=(
        "streaming: frame‑by‑frame with KV cache; "
        "windowed: overlapping windows for long sequences"
    )
)

```

The dispatch logic around lines **[640-660](https://github.com/Robbyant/lingbot-map/blob/main/demo.py#L640-L660)** forwards the flag to the appropriate method:

```python
if args.mode == "streaming":
    predictions = model.inference_streaming(...)
else:
    predictions = model.inference_windowed(...)

```

## Streaming Mode: Frame-by-Frame with KV Cache

The **streaming mode** processes sequences incrementally while storing compressed attention maps in a KV cache. This approach minimizes per-frame GPU memory to approximately the size of a single frame plus the growing cache.

### Implementation Details

The core implementation resides in [[`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py)](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py). The `inference_streaming` method (lines **[49-73](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py#L49-L73)**) executes three phases:

1. **Scale-frame initialization** – Processes the first `num_scale_frames` (default 8) together using bidirectional attention via a special scale token.
2. **KV cache cleanup** – Calls `self.clean_kv_cache()` to reset state from previous runs.
3. **Per-frame streaming loop** – Iterates through remaining frames (lines **[447-493](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py#L447-L493)**), checking `keyframe_interval` to determine whether to store the frame in the KV cache or skip appending via `_set_skip_append(True)`.

```python
def inference_streaming(
    self,
    images: torch.Tensor,
    num_scale_frames: Optional[int] = None,
    keyframe_interval: int = 1,
    output_device: Optional[torch.device] = None,
) -> Dict[str, torch.Tensor]:
    # Scale-frame phase

    scale_output = self.forward(scale_images, ...)
    # Streaming loop

    for i in range(scale_frames, S):
        is_keyframe = (keyframe_interval <= 1) or ((i - scale_frames) % keyframe_interval == 0)
        if not is_keyframe:
            self._set_skip_append(True)
        frame_output = self.forward(frame_image, ...)
        if not is_keyframe:
            self._set_skip_append(False)
        # Accumulate predictions...

```

### Memory and Performance Characteristics

- **GPU memory**: Scales linearly with the number of stored keyframes, not the total sequence length.
- **Latency**: Extremely low per-frame delay since only the current frame resides in active GPU memory.
- **Throttling**: Increase `--keyframe_interval` (e.g., to 6) to store only every Nth frame, drastically reducing cache size at the cost of minor accuracy degradation.

## Windowed Mode: Overlapping Windows for Long Sequences

The **windowed mode** splits long videos into overlapping chunks, processes each with a fresh KV cache, then aligns and stitches the results. This is essential for sequences where a single streaming run would exhaust GPU memory.

### Window Geometry and Alignment

Implemented in [[`lingbot_map/models/gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py)](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py), the `inference_windowed` method (lines **[59-73](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py#L59-L73)**) calculates window geometry at lines **[613-639](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py#L613-L639)**:

- **`window_size`**: Number of keyframes per window (including scale frames).
- **`overlap_keyframes`**: Determines how many frames duplicate between consecutive windows to ensure smooth transitions.

The algorithm processes each window independently (lines **[682-735](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py#L682-L735)**), then performs `_pairwise_alignment` and `_stitch_windows` (lines **[822-860](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py#L822-L860)**) to merge predictions into a single continuous trajectory.

### When to Use Windowed Inference

- **Very long videos** (≥500 frames or minutes of footage) where streaming KV cache would exceed GPU memory.
- **High-quality global reconstruction** requiring drift correction across the entire sequence.
- **Bounded memory**: Peak GPU usage is capped by `window_size` rather than growing with video length.

## Practical Usage Examples

### Streaming Example with Keyframe Interval

For short-to-moderate sequences with reduced memory footprint:

```bash
python examples/demo.py \
    --model_path checkpoints/gct.pt \
    --image_folder data/seq1/ \
    --mode streaming \
    --keyframe_interval 6 \
    --offload_to_cpu

```

This stores only every 6th frame in the KV cache while processing all frames.

### Windowed Example for Long Videos

For 2-minute drone footage at 10 FPS:

```bash
python examples/demo.py \
    --model_path checkpoints/gct.pt \
    --video_path data/flight.mp4 \
    --fps 10 \
    --mode windowed \
    --window_size 64 \
    --overlap_keyframes 8 \
    --keyframe_interval 2

```

Here, `--window_size 64` limits each chunk to 64 keyframes, while 8 overlapping keyframes ensure smooth alignment between windows.

### Direct Python API Usage

You can invoke either mode directly from Python:

```python
import torch
from lingbot_map.models.gct_stream import GCTStream

# Load model

model = GCTStream(
    img_size=518,
    patch_size=14,
    enable_3d_rope=True,
    num_scale_frames=8,
).eval().to("cuda")

# Streaming inference

pose_stream = model.inference_streaming(
    images,
    num_scale_frames=8,
    keyframe_interval=1,
    output_device=torch.device("cpu")
)

# Windowed inference

pose_windowed = model.inference_windowed(
    images,
    window_size=64,
    overlap_keyframes=8,
    keyframe_interval=2,
    output_device=torch.device("cpu")
)

```

## Summary

- **Streaming mode** (`--mode streaming`) maintains a persistent KV cache for frame-by-frame processing, offering the lowest latency and linear memory growth suitable for short-to-moderate sequences.
- **Windowed mode** (`--mode windowed`) partitions sequences into overlapping windows with independent KV caches, providing bounded GPU memory for long videos at the cost of additional alignment overhead.
- Both modes share the `GCTStream` architecture and support `--keyframe_interval` to trade memory for accuracy by skipping KV cache updates on non-critical frames.
- Use `--offload_to_cpu` in either mode to move predictions to system RAM immediately after computation.

## Frequently Asked Questions

### How do I choose between streaming and windowed modes in LingBot-Map?

**Choose `--mode streaming`** when processing short sequences (under ~500 frames) or when you need minimal latency per frame, such as real-time robot perception. **Choose `--mode windowed`** when processing long videos that would exceed GPU memory with a single KV cache, or when you require global alignment across the entire sequence to minimize drift.

### What is the KV cache and why does it affect memory usage?

The **KV cache** (key-value cache) stores compressed attention activations from previous frames to enable causal processing without recomputing past states. In streaming mode, this cache grows with each keyframe, potentially consuming large GPU memory for long videos. Windowed mode resets this cache for each window, bounding memory usage to the window size.

### Can I adjust memory usage without switching modes?

Yes. In **streaming mode**, increase `--keyframe_interval` to store only every Nth frame in the cache (e.g., `--keyframe_interval 6`). In **windowed mode**, reduce `--window_size` to process smaller chunks. Both approaches reduce GPU memory at the cost of potential accuracy degradation due to fewer stored attention states.

### Does windowed mode produce lower quality results than streaming?

Windowed mode typically produces **higher quality** for very long sequences because the alignment step (`_pairwise_alignment`) corrects drift between windows. However, it requires an extra stitching computation, making it slightly slower than pure streaming for short sequences where memory is not a constraint.