# How to Address Pose Collapse in LingBot-Map for Very Long Sequences > Fix pose collapse in LingBot-Map for long sequences. Learn to bound KV cache size and prevent RoPE drift with windowed inference and keyframe selection. - Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map) - Tags: how-to-guide - Published: 2026-06-29 --- **Use windowed inference with keyframe selection to bound the KV cache size and prevent 3-D RoPE positional encoding drift.** LingBot-Map predicts camera poses frame-by-frame while maintaining a **KV cache** that stores attention keys and values from previously seen frames. When processing videos significantly longer than the training data (~320 frames), the cache grows beyond the model's 3-D Rotary Positional Embedding (RoPE) temporal range, causing estimated poses to drift—a phenomenon documented as **pose collapse**. This guide explains how to address pose collapse in LingBot-Map for very long sequences using the windowed inference and keyframe selection mechanisms implemented in the Robbyant/lingbot-map repository. ## Understanding Pose Collapse in LingBot-Map Pose collapse occurs due to two interacting architectural constraints in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py): - **3-D RoPE Limitations**: The model uses 3-D Rotary Positional Embedding to encode temporal information, assuming a maximum frame index (`max_frame_num`, default `1024`). When the KV cache stores more frames than seen during training, the positional encoding becomes out-of-range, and learned attention patterns fail to align correctly. - **Unbounded KV Cache Growth**: In standard streaming mode (`--mode streaming`), the cache grows linearly with frame count. For videos exceeding 3000 frames, this quickly surpasses the trained RoPE range, resulting in accumulated pose drift. ## Windowed Inference: The Primary Solution The **windowed inference** mechanism processes video in overlapping segments, resetting the KV cache at the start of each window to keep memory bounded. In `GCTStream.inference_windowed` (lines 1023-1039), each window receives a fresh KV cache limited by `window_size` (measured in keyframes, not raw frames). This guarantees that the cache never exceeds the RoPE temporal space regardless of total video length. ### Cache Management and Window Stitching When a window completes, the model must align its predictions to the previous window to maintain global consistency. The repository implements this through: - **`_pairwise_alignment`** (line 1065): Computes a similarity transform (scale-rotation-translation) between overlapping regions. - **`_warp_predictions`**: Applies this transform to stitch windows together without drift. - **`_stitch_windows`** (line 1045): Orchestrates the alignment using `overlap_keyframes` (or `overlap_size`) to ensure at least one paired keyframe exists for robust alignment. This stitching process ensures that while each window operates with an independent, bounded cache, the final trajectory remains globally consistent. ## Keyframe Selection Strategies Beyond windowing, LingBot-Map provides mechanisms to reduce KV cache growth within each window by storing attention values only for selected frames. ### Fixed Interval Keyframes The `keyframe_interval` parameter (default 1) specifies that only every *N*-th frame retains its KV cache entry. In `inference_streaming` (lines 1064-1069), non-keyframes invoke `_set_skip_append(True)`, which prevents appending to the cache while still allowing pose prediction. Setting `--keyframe_interval 4` reduces cache memory usage by approximately 75% without requiring optical flow computation. ### Flow-Based Adaptive Keyframes For scenes with variable motion, the `--flow_threshold` flag enables dynamic keyframe selection. The `_compute_flow_magnitude` function (lines 92-124) measures optical flow between the current frame and the last keyframe. When mean flow exceeds the threshold (in pixels), the frame automatically becomes a keyframe regardless of the fixed interval. This ensures fast-motion sections retain sufficient temporal context while static sections benefit from aggressive pruning. ## Code Examples: Running Long Sequence Inference The following commands demonstrate how to configure LingBot-Map for arbitrarily long sequences using the windowed inference pipeline: ### Basic Windowed Inference ```bash python demo.py \ --model_path /path/to/lingbot-map-long.pt \ --video_path video.mp4 \ --fps 10 \ --mode windowed \ --window_size 128 ``` ### Reduced Cache Growth with Fixed Intervals ```bash python demo.py \ --model_path /path/to/lingbot-map-long.pt \ --video_path video.mp4 \ --fps 10 \ --mode windowed \ --window_size 128 \ --overlap_keyframes 16 \ --keyframe_interval 4 ``` ### Flow-Based Adaptive Keyframes ```bash python demo.py \ --model_path /path/to/lingbot-map-long.pt \ --video_path video.mp4 \ --fps 10 \ --mode windowed \ --window_size 128 \ --overlap_keyframes 16 \ --flow_threshold 2.0 ``` ### CPU Offloading for Extremely Long Sequences ```bash python demo.py \ --model_path /path/to/lingbot-map-long.pt \ --video_path video.mp4 \ --fps 10 \ --mode windowed \ --window_size 128 \ --overlap_keyframes 16 \ --keyframe_interval 3 \ --output_device cpu ``` ## Summary - **Pose collapse** results from 3-D RoPE encoding limitations when the KV cache exceeds the trained temporal range (~320 frames, max 1024). - **Windowed inference** (`--mode windowed`) in [`gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream_window_v2.py) bounds cache size by processing video in overlapping segments with fresh caches. - **Window stitching** via `_pairwise_alignment` and `_warp_predictions` maintains global trajectory consistency across segments. - **Keyframe selection** through `--keyframe_interval` or `--flow_threshold` further reduces cache growth by skipping KV storage for non-essential frames. - Combining these mechanisms allows LingBot-Map to process tens of thousands of frames without pose drift. ## Frequently Asked Questions ### What exactly causes pose collapse in LingBot-Map? Pose collapse occurs when the KV cache grows larger than the 3-D RoPE positional encoding can represent (default `max_frame_num` of 1024). Since the model was trained on sequences of approximately 320 frames, attention patterns become unreliable when the cache contains significantly more frames, causing estimated camera poses to drift progressively. ### How does windowed inference prevent pose drift? Windowed inference resets the KV cache at the start of each window (controlled by `--window_size`), ensuring the cache never exceeds the RoPE range. The `_stitch_windows` method aligns consecutive windows using a similarity transform computed from overlapping keyframes, effectively "re-zeroing" the positional encoding without losing trajectory continuity. ### What is the difference between keyframe_interval and flow_threshold? `--keyframe_interval` uses a fixed temporal subsampling (every Nth frame), while `--flow_threshold` enables adaptive selection based on optical flow magnitude. Fixed intervals provide predictable memory usage, whereas flow-based selection preserves more keyframes during rapid camera motion and fewer during static scenes, optimizing cache efficiency dynamically. ### Where is the windowed inference logic implemented? The core implementation resides in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py), specifically within the `GCTStream.inference_windowed` method (lines 1023-1039). Supporting utilities including `_pairwise_alignment` (line 1065), `_warp_predictions`, and `_compute_flow_magnitude` (lines 92-124) are defined in the same file.