Why Pose Collapse Occurs at Long Distances and How Windowed Mode Solves It in LingBot-Map

Pose collapse happens when LingBot-Map's Geometric Context Transformer processes sequences beyond its RoPE training range (≤320 frames), causing the KV cache to overflow and positional encodings to degrade; windowed mode prevents this by segmenting video into bounded windows with periodic cache resets and cross-window geometric alignment.

LingBot-Map predicts camera poses using a Geometric Context Transformer (GCT) that maintains a temporal key-value (KV) cache for attention across video frames. While this design works for short sequences, inference beyond the training distance triggers catastrophic geometric drift. The repository implements a specialized windowed mode to handle arbitrarily long videos without sacrificing pose accuracy.

What Causes Pose Collapse at Long Distances

Pose collapse is a degenerative failure mode where the model's predictions gradually become inconsistent and collapse to invalid configurations. Two mechanisms drive this failure when processing long sequences:

KV-Cache Overflow Beyond Training Range

The GCT model is trained on video sequences containing ≤320 frames, which defines its Rotary Position Embedding (RoPE) training range. During inference, the KV cache grows monotonically as each new frame is processed. According to the repository documentation at README.md line 241, when the sequence length exceeds this trained boundary, "the attention patterns no longer have a reliable positional encoding."

Geometric Drift from Unchecked Accumulation

Without positional anchors, the transformer cannot maintain geometric consistency between the current view and earlier frames. The model loses track of spatial relationships, causing pose predictions to drift until they collapse into geometrically impossible configurations. This manifests as sudden jumps or gradual degradation in the camera trajectory when processing long-distance walkthroughs.

How Windowed Mode Prevents Pose Collapse

Windowed inference processes video in overlapping segments rather than as a single continuous stream. This approach bounds the effective history seen by the transformer, keeping positional encodings within the learned regime.

Cache Resetting and Window Segmentation

Each window begins with a complete cache reset. In lingbot_map/models/gct_stream_window_v2.py, the clean_kv_cache method (lines 88-99) removes all stored KV pairs at the start of every window. This limits the attention context to window_size keyframes (default 128), well within the 320-frame training limit.

Scale Frames for Reference Establishment

At each window boundary, the system processes 8 scale frames bidirectionally to establish a common coordinate system. These frames provide a stable geometric reference before processing the remaining window content, ensuring local consistency regardless of absolute distance from the video start.

Keyframe Sampling for Memory Efficiency

Rather than caching every frame, the system uses _set_skip_append (lines 64-69) to implement keyframe sampling. Only every N-th frame (controlled by keyframe_interval) is stored in the KV cache. Non-keyframes attend to cached context but are not appended, reducing memory usage by half or more depending on the interval setting.

Cross-Window Alignment and Stitching

Adjacent windows share an overlap region (either actual frames or keyframes). After each window completes, _pairwise_alignment (lines 842-880) estimates a scaled similarity transform (scale s, rotation R, translation t) on the overlapping region. The _warp_predictions method applies this transform to align the later window with the previous trajectory before _stitch_windows concatenates the results. This geometric stitching maintains global consistency while preserving local window boundaries.

Configuring Windowed Inference

The windowed logic is exposed through both command-line tools and programmatic APIs in lingbot_map/models/gct_stream_window_v2.py.

Interactive Demo Usage

For real-time processing with visualization, use demo.py with the --mode windowed flag:

python demo.py \
    --model_path /path/to/lingbot-map.pt \
    --video_path video.mp4 \
    --fps 10 \
    --mode windowed \
    --window_size 128 \
    --overlap_keyframes 16 \
    --keyframe_interval 2

• --window_size 128 limits each window to 128 keyframes (approximately 120 actual frames when using --keyframe_interval 2).
• --overlap_keyframes 16 ensures 16 keyframes overlap between windows for robust alignment.
• --keyframe_interval 2 stores only every second frame in the KV cache, halving memory consumption.

Batch Processing Long Videos

For headless rendering of very long sequences (e.g., 25,000 frames), use demo_render/batch_demo.py:

python demo_render/batch_demo.py \
    --video_path /data/indoor_travel.MP4 \
    --output_folder /data/outputs/indoor/ \
    --model_path /path/to/lingbot-map.pt \
    --config demo_render/config/indoor.yaml \
    --mode windowed \
    --window_size 128 \
    --keyframe_interval 10 \
    --overlap_keyframes 8

This configuration processes the video via model.inference_windowed without GUI overhead, using a keyframe interval of 10 for extremely long sequences.

Programmatic Python API

For custom pipelines, call inference_windowed directly:

import torch
from lingbot_map.models.gct_stream_window_v2 import GCTStream

model = GCTStream(...)
images = torch.rand(1, 25000, 3, 518, 378)  # [B, S, C, H, W]

pred = model.inference_windowed(
    images,
    window_size=128,
    overlap_keyframes=8,
    keyframe_interval=2,
    flow_threshold=0.0,  # Uses fixed-interval mode

)

The method returns a dictionary containing concatenated pose encodings, depth maps, and point clouds that are already aligned across windows via the internal _pairwise_alignment logic.

Summary

• Pose collapse occurs when LingBot-Map's KV cache exceeds the 320-frame RoPE training range, causing unreliable positional encodings and geometric drift.
• Windowed mode prevents collapse by segmenting sequences into bounded windows (default 128 keyframes) and resetting the KV cache via clean_kv_cache at each window boundary.
• Keyframe sampling (_set_skip_append) reduces memory usage by only caching every N-th frame while maintaining temporal attention.
• Cross-window alignment uses _pairwise_alignment and _warp_predictions to apply similarity transforms on overlapping regions, stitching windows into a globally consistent trajectory.
• The implementation resides in lingbot_map/models/gct_stream_window_v2.py, with entry point inference_windowed (lines 1022-1026).

Frequently Asked Questions

What exactly causes pose collapse in LingBot-Map?

Pose collapse occurs when the Geometric Context Transformer's KV cache grows beyond the 320-frame limit seen during training. Beyond this range, the RoPE positional encodings become unreliable, causing the model to lose geometric anchor points between frames and drift into degenerate pose predictions.

How does windowed mode differ from standard inference?

Standard inference processes the entire video as a single stream with a monotonically growing KV cache. Windowed mode segments the video into independent chunks, clears the cache between windows via clean_kv_cache, and uses _pairwise_alignment to geometrically stitch chunks together. This bounds the attention context to the training range while maintaining global trajectory consistency.

Can I adjust the window size for my specific hardware constraints?

Yes. The --window_size parameter controls how many keyframes are processed before resetting the cache. While the default is 128, you can reduce this value for GPUs with limited memory or increase it up to the 320-frame training limit if processing power allows. Note that --overlap_keyframes must be set sufficiently high (typically 8-16) to ensure reliable alignment between windows.

Why are scale frames necessary at the start of each window?

Scale frames provide a bidirectional geometric reference that anchors the window's coordinate system. Because the KV cache is cleared between windows, these 8 frames establish the initial scale and pose reference before processing the remaining keyframes, preventing discontinuities at window boundaries.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}