When to Use GCTStreamWindow vs GCTStream for Your Dataset

Choose GCTStream for short-to-moderate sequences (≤200 frames) where full temporal context improves accuracy, and switch to GCTStreamWindow when processing long videos or deploying in memory-constrained environments that require constant O(window) memory usage.

The lingbot-map repository by Robbyant provides streaming variants of the Global Context Transformer (GCT) designed for video pose estimation and depth prediction. Both GCTStream and GCTStreamWindow process frames incrementally using a key-value (KV) cache to avoid recomputing attention over the entire sequence, but they differ fundamentally in how they manage temporal context and memory consumption.

Understanding the GCT Streaming Architecture

Both classes extend the base GCT architecture to support incremental inference. According to the source code in lingbot_map/aggregator/stream.py, they leverage AggregatorStream to maintain a KV cache that persists across forward passes. This allows the model to process video streams frame-by-frame without storing the entire sequence in GPU memory simultaneously.

However, the critical distinction lies in the KV cache eviction policy. While GCTStream (defined in lingbot_map/models/gct_stream.py) retains all historical key-value pairs, GCTStreamWindow (in lingbot_map/models/gct_stream_window.py) implements a sliding-window mechanism that discards older frames to maintain constant memory usage.

KV Cache Management: The Critical Difference

GCTStream: Full Causal Attention

In GCTStream, the KV cache grows linearly with sequence length (O(S)). The class constructs its cache using AggregatorStream with use_sdpa=False to enable the FlashInfer backend, but never evicts old KV blocks. This provides full causal attention where every past frame remains attendable, maximizing temporal consistency for pose refinement and depth estimation.

GCTStreamWindow: Sliding Window Attention

GCTStreamWindow overrides this behavior by passing kv_cache_sliding_window and related flags to the underlying AggregatorStream. As implemented in lingbot_map/models/gct_stream_window.py, this bounds memory usage to O(window) regardless of video length. The cache retains only the most recent N KV blocks (default 64) plus optional scale frames configured via kv_cache_scale_frames and kv_cache_cross_frame_special.

When to Use GCTStream for Maximum Temporal Context

Prefer GCTStream when your dataset contains short-to-moderate sequences (approximately ≤200 frames) and GPU memory permits linear growth. This approach is ideal when:

• You require the strongest possible temporal consistency across the entire video
• The scene complexity benefits from long-range attention (e.g., loop closure detection in visual odometry)
• You can afford the O(S) memory footprint during inference

Note that while GCTStream accepts kv_cache_sliding_window in its constructor, this parameter is ignored and the cache never evicts.

When to Use GCTStreamWindow for Long Video Streaming

Switch to GCTStreamWindow when processing very long videos (thousands of frames) or deploying in low-memory environments such as single-GPU inference servers. This variant excels when:

• Video length would otherwise cause out-of-memory (OOM) errors with full caching
• Scene dynamics are locally consistent (recent context suffices)
• You need predictable, constant memory usage independent of sequence length

The bounded memory model allows processing hour-long recordings by tuning kv_cache_sliding_window (typically 64–128) to retain sufficient recent context while evicting older frames.

Code Examples: Configuring Each Streaming Model

Full Context Streaming with GCTStream

from lingbot_map.models.gct_stream import GCTStream

# Optimal for videos ~150 frames

model = GCTStream(
    img_size=518,
    patch_size=14,
    embed_dim=1024,
    sliding_window_size=-1,          # Full causal attention

    enable_stream_inference=True,
    kv_cache_sliding_window=64,      # Ignored - cache never evicts

)

# Process frame-by-frame with KV cache accumulation

preds = model.inference_streaming(video_tensor, keyframe_interval=1)

Sliding Window Streaming with GCTStreamWindow

from lingbot_map.models.gct_stream_window import GCTStreamWindow

# Optimal for long videos ~3000+ frames

model = GCTStreamWindow(
    img_size=518,
    patch_size=14,
    embed_dim=1024,
    sliding_window_size=-1,          # Feed full sequence to model

    kv_cache_sliding_window=64,      # Retain only last 64 KV blocks

    kv_cache_scale_frames=8,         # Keep 8 scale frames for stability

    kv_cache_cross_frame_special=True,
    enable_stream_inference=True,
)

# Memory stays constant regardless of video length

preds = model.inference_streaming(video_tensor, keyframe_interval=5)

In the second example, GPU memory consumption remains roughly constant at 64 × (embedding_dim × patches) bytes even for thousands of frames, whereas the first example scales linearly with frame count.

Source Code Reference

The implementation differences between these streaming variants are located in the following files:

• lingbot_map/models/gct_stream.py – Core streaming model with unbounded KV cache and full causal attention.
• lingbot_map/models/gct_stream_window.py – Adds sliding-window eviction via kv_cache_sliding_window configuration.
• lingbot_map/models/gct_stream_window_v2.py – Refined implementation introducing additional cache management flags.
• lingbot_map/aggregator/stream.py – Contains AggregatorStream, the base component managing KV cache storage and FlashInfer backend integration.
• lingbot_map/heads/camera_head.py – Provides the causal camera head utilized by both streaming architectures.

Summary

• Use GCTStream when processing ≤200 frames with available GPU memory, prioritizing full temporal context and maximum accuracy.
• Use GCTStreamWindow when processing long videos or deploying with memory constraints, accepting bounded context for constant O(window) memory usage.
• Key parameter: kv_cache_sliding_window controls the window size (default 64) in GCTStreamWindow but is ignored in GCTStream.
• Implementation: Both rely on AggregatorStream in lingbot_map/aggregator/stream.py, differing only in cache eviction policy.

Frequently Asked Questions

Does GCTStreamWindow lose accuracy compared to GCTStream?

GCTStreamWindow may exhibit slightly reduced long-range temporal consistency since it cannot attend to frames outside the sliding window. However, for slowly changing scenes or when combined with kv_cache_scale_frames to retain key reference frames, the accuracy loss is minimal compared to the memory savings achieved.

What is the default KV cache size in GCTStreamWindow?

According to the source code in lingbot_map/models/gct_stream_window.py, the default kv_cache_sliding_window is 64 blocks. You can increase this to 128 or higher if GPU memory allows and your application requires longer temporal context, or reduce it further to minimize memory footprint.

Can I switch between GCTStream and GCTStreamWindow without retraining?

Yes, both classes share the same underlying architecture and weights. They are inference-time variants that differ only in KV cache management. You can initialize either model from the same checkpoint and switch based on your deployment constraints without modifying the trained parameters.

Why does GCTStream accept kv_cache_sliding_window if it doesn't use it?

The parameter exists in GCTStream for API compatibility with the base streaming interface, but as implemented in lingbot_map/models/gct_stream.py, the value is ignored and the cache maintains all historical frames. Only GCTStreamWindow and its variants in gct_stream_window.py and gct_stream_window_v2.py actually utilize this parameter for cache eviction.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}