# How to Debug Pose Collapse or Drift Issues in Long Sequences Using LingBot-Map

> Debug pose collapse and drift in long sequences using LingBot-Map. Monitor KV cache, set kv_cache_sliding_window, and verify image_size_hw for stable results.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: how-to-guide
- Published: 2026-07-08

---

**Set the environment variable `LINGBOT_DEBUG_KV=1` to monitor KV-cache occupancy, ensure `kv_cache_sliding_window` is large enough for your sequence length, and verify that `image_size_hw` matches your actual frame dimensions to eliminate pose collapse and drift in long video sequences.**

LingBot-Map processes video sequences frame-by-frame using a streaming transformer that caches key-value (KV) pairs for causal attention. When handling very long sequences, developers frequently encounter **pose collapse** (where the predicted camera pose converges to a constant value, often the origin) or **pose drift** (where the trajectory gradually deviates from ground truth). These issues stem from how the KV cache manages memory, how keyframes are selected, and how field-of-view (FoV) parameters are reconstructed from the pose encoding.

## Understanding the Streaming Architecture

LingBot-Map’s architecture consists of three primary components that handle long sequences:

- **`GCTStream`** – The top-level model in [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py) that orchestrates the forward pass, manages KV-cache cleaning, and classifies frames as either keyframes (KV kept) or non-keyframes (KV discarded). The `inference_streaming` method serves as the main entry point for streaming inference, while `_set_skip_append` controls whether a frame’s KV pairs are stored.

- **`AggregatorStream`** – Located in [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py), this module builds transformer blocks, initializes the paged KV cache (supporting FlashInfer or SDPA), and optionally applies 3-D rotary position embeddings (RoPE) for temporal consistency. The `_init_kv_cache` method allocates cache memory, and `clean_kv_cache` resets it between sequences.

- **`CameraCausalHead`** – Predicts the **pose encoding** (`absT_quaR_FoV`) from aggregated tokens. The encoding is converted to full extrinsic/intrinsic camera parameters via `pose_encoding_to_extri_intri` in [`lingbot_map/utils/pose_enc.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/pose_enc.py).

Error metrics are computed using `se3_to_relative_pose_error` in the same file, which calculates relative rotation and translation angles between predicted and ground-truth SE(3) matrices.

## Root Causes of Pose Collapse and Drift

Pose collapse and drift typically originate from five specific areas in the codebase:

**KV-Cache Overflow**

The `kv_cache_sliding_window` parameter (default 64) determines how many past frames remain in cache. If your sequence exceeds this window without keyframe pruning, the model attends only to recent frames, causing sudden collapse. Check this in `AggregatorStream.__init__` and `_init_kv_cache` in [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py).

**Stale Scale Frames**

The first `num_frame_for_scale` frames require bidirectional attention to establish scale. If `keyframe_interval` is set greater than 1 before these frames complete processing, the scale token may never see all required frames, leading to unstable FoV estimates. This logic resides in `GCTStream.inference_streaming` around lines 514-528.

**Keyframe Logic Errors**

Non-keyframes must have `_skip_append=True` to prevent their KV pairs from being cached. A bug that forgets to reset this flag causes every frame to be cached, exhausting memory and triggering collapse. Verify this in `GCTStream._set_skip_append`.

**Incorrect Field-of-View Reconstruction**

The FoV values (`fov_h`, `fov_w`) derive from intrinsics and `image_size_hw`. If height and width are swapped, the resulting intrinsics distort, propagating drift through the pose encoding. Inspect `extri_intri_to_pose_encoding` in [`lingbot_map/utils/pose_enc.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/pose_enc.py).

**3-D RoPE Misalignment**

When 3-D RoPE is enabled, global frame indices (`f_start`, `f_end`) must remain consistent across blocks. Mismatches give tokens incorrect temporal offsets, causing gradual drift. Check `_get_3d_positions_streaming` and `_process_causal_stream` in [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py).

## Step-by-Step Debugging Workflow

Follow this exact sequence to isolate the failure mode:

1. **Enable KV-cache logging** – Set `LINGBOT_DEBUG_KV=1` before running. The `_log_kv_stats` helper in [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py) prints per-frame cache occupancy, page usage, and special-token counts.

2. **Run a short test** – Process 10 frames and verify that cached blocks grow linearly with `kv_cache_sliding_window` and that memory estimates remain stable.

3. **Increase sequence length gradually** – When pose output suddenly flattens, inspect KV stats. If `frame_count` stops increasing while sequence length grows, you are experiencing cache eviction. Reduce `kv_cache_sliding_window` or increase `kv_cache_scale_frames`.

4. **Validate FoV** – After forward pass, recover intrinsics with `pose_encoding_to_extri_intri` and compare to ground truth. Large discrepancies indicate incorrect `image_size_hw`.

5. **Compute drift** – Call `se3_to_relative_pose_error(pred_se3, gt_se3, num_frames)` on your predicted and ground-truth SE(3) matrices. Plot rotation and translation angles over time; monotonic increases point to cache eviction or non-keyframe skip issues.

## Practical Implementation

This complete example demonstrates streaming inference with debugging enabled:

```python
import os
import torch
from lingbot_map.models.gct_stream import GCTStream
from lingbot_map.utils.pose_enc import se3_to_relative_pose_error

# Build the model with FlashInfer KV cache

model = GCTStream(
    img_size=518,
    patch_size=14,
    embed_dim=1024,
    enable_camera=True,
    enable_3d_rope=False,          # typically off for debugging

    kv_cache_sliding_window=64,     # default

    kv_cache_scale_frames=8,
    enable_stream_inference=True,
)

model.eval()
model.to("cuda")

# Prepare 120-frame demo sequence

images = torch.randn(1, 120, 3, 518, 518).cuda()

# Enable per-frame KV stats

os.environ["LINGBOT_DEBUG_KV"] = "1"

# Run streaming inference with keyframe interval

out = model.inference_streaming(
    images,
    num_scale_frames=4,        # first 4 frames are scale frames

    keyframe_interval=5,       # keep every 5th frame in KV cache

    output_device=torch.device("cpu")
)

# Clean KV cache before next video

model.clean_kv_cache()

# Compute drift against ground truth

gt_se3 = torch.eye(4).unsqueeze(0).repeat(120, 1, 1)  # example ground truth

pred_se3 = out["pose_enc"]  # shape [B, S, 9]

rot_err, trans_err = se3_to_relative_pose_error(
    pred_se3, gt_se3, num_frames=out["pose_enc"].shape[1]
)

print(f"Mean rotation drift: {rot_err.mean().item():.2f} deg")
print(f"Mean translation drift: {trans_err.mean().item():.2f} deg")

```

## Quick Parameter Reference

Use this guide to match symptoms with corrective actions:

| Symptom | Action |
|---------|--------|
| **Rapid collapse after ~64 frames** | Increase `kv_cache_sliding_window` or `kv_cache_scale_frames`. If memory is limited, raise `keyframe_interval` to 2 or higher. |
| **Gradual drift despite sufficient cache** | Verify `image_size_hw` matches actual frame dimensions; an off-by-one swap corrupts FoV reconstruction. |
| **Memory OOM on long videos** | Set `output_device=torch.device('cpu')` to offload predictions per frame, and/or lower `kv_cache_sliding_window`. |
| **Non-keyframes still cached** | Ensure `_set_skip_append(True)` is called for non-keyframes in `inference_streaming`. Add manual `model._set_skip_append(True)` after the loop to verify. |

## Summary

- **Enable diagnostics** by setting `LINGBOT_DEBUG_KV=1` to monitor KV-cache occupancy in real-time.
- **Size the cache** correctly by ensuring `kv_cache_sliding_window` accommodates your sequence length or by increasing `keyframe_interval` to reduce memory pressure.
- **Clean between sequences** by calling `model.clean_kv_cache()` to prevent stale context from bleeding into new videos.
- **Validate FoV settings** by checking that `image_size_hw` matches your actual frame dimensions to prevent distortion in `pose_encoding_to_extri_intri`.
- **Measure drift quantitatively** using `se3_to_relative_pose_error` to distinguish between sudden collapse and gradual divergence.

## Frequently Asked Questions

### Why does the pose collapse after exactly 64 frames?

This occurs when the default `kv_cache_sliding_window=64` fills completely and begins evicting older frames. Once the cache can only see the most recent frames, the model loses temporal context and collapses to a constant pose. Increase `kv_cache_sliding_window` or implement a `keyframe_interval` greater than 1 to keep only essential frames in cache.

### How do I clean the KV cache between different video sequences?

Call `model.clean_kv_cache()` after processing each video. This method is also invoked automatically at the start of `inference_streaming`, but explicit cleaning ensures no residual KV pairs from previous sequences contaminate new predictions. Failure to clean the cache causes drift as the model attends to unrelated historical frames.

### What is the relationship between keyframe_interval and KV cache memory usage?

The `keyframe_interval` parameter determines how many frames pass between each cached keyframe. Setting `keyframe_interval=5` means only every fifth frame is stored, reducing cache occupancy by 80%. This trade-off allows processing of very long sequences with limited GPU memory, though overly sparse keyframes may reduce accuracy in fast-motion scenes.

### How can I verify if my FoV reconstruction is causing drift?

After inference, extract the predicted intrinsics using `pose_encoding_to_extri_intri(pred_pose_enc)` and compare the principal point coordinates to your image center (`W/2`, `H/2`). If the values diverge significantly, verify that `image_size_hw` was passed as `(height, width)` rather than `(width, height)` when constructing the model. Incorrect dimensions distort the FoV calculation in `extri_intri_to_pose_encoding`, propagating geometric errors through the entire sequence.