# Common Failure Modes When Running LingBot-Map on New Datasets and How to Debug Them

> Debug common LingBot-Map failures on new datasets including KV-cache issues, pose drift, and format mismatches. Learn solutions with gct_profile.py.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: how-to-guide
- Published: 2026-07-04

---

**LingBot-Map failures on new datasets typically stem from KV-cache memory exhaustion beyond the 320-frame training limit, pose drift in the Geometric Context Transformer's RoPE embeddings, or strict dataset format mismatches, all diagnosable via [`gct_profile.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_profile.py) and resolvable through keyframe interval adjustments, windowed inference, or preprocessing pipelines.**

When deploying the **LingBot-Map** 3D foundation model on custom video sequences or unprocessed benchmarks, practitioners frequently encounter specific **common failure modes when running on new datasets** that stall inference or corrupt geometric reconstructions. According to the `Robbyant/lingbot-map` source code, these failures originate in the **Geometric Context Transformer** (GCT) architecture—specifically within the paged KV-cache in [`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py), the rotary positional embeddings in [`lingbot_map/layers/rope.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/rope.py), and the preprocessing logic required for input integrity. This guide provides concrete debugging strategies using the actual implementation to restore stable performance.

## Understanding the Core Architecture

The GCT architecture relies on several interconnected components that become failure points when datasets deviate from training expectations. The **Vision Transformer** in [`lingbot_map/layers/vision_transformer.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/vision_transformer.py) processes image patches, while **RoPE** ([`lingbot_map/layers/rope.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/rope.py)) injects relative pose information into attention mechanisms. The **paged KV-cache** ([`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py)) enables streaming inference but grows linearly with keyframes. Finally, the preprocessing pipeline ([`preprocess/oxford.py`](https://github.com/Robbyant/lingbot-map/blob/main/preprocess/oxford.py)) enforces strict naming and metadata conventions required by [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py).

## Three Common Failure Modes When Running on New Datasets

### Memory-Related Issues and KV-Cache Exhaustion

The KV-cache grows linearly with the number of stored keyframes. By default, LingBot-Map treats every frame as a keyframe, quickly exceeding the **320-frame RoPE training range** and triggering out-of-memory (OOM) errors.

Typical symptoms include sudden GPU memory spikes, long stalls between frames, or abrupt FPS drops. To debug, first verify that **FlashInfer** is installed and selected as the backend, as it provides optimized paged memory management compared to SDPA.

Run the profiler to inspect cache occupancy:

```bash
python gct_profile.py --backend flashinfer --dtype bf16 --compile

```

If the output shows KV-cache slots approaching 100%, reduce the storage frequency:

```bash
python demo.py \
    --model_path /path/to/lingbot-map-long.pt \
    --image_folder /data/new_dataset \
    --keyframe_interval 2 \
    --use_flashinfer

```

For sequences exceeding several thousand frames, switch to **windowed mode** (`--mode windowed --window_size 128`) to reset the cache periodically.

### Pose-Collapse and Drift Beyond 320 Frames

The model was trained on sequences of approximately **320 frames**. Beyond this limit, the relative pose embeddings in [`lingbot_map/layers/rope.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/rope.py) become unstable, especially when the KV-cache contains non-keyframes or excessive history.

Symptoms manifest as sudden trajectory jumps in the point-cloud viewer, blurry reconstructions, or duplicated geometry.

Enable **windowed inference** to force cache resets:

```bash
python demo.py \
    --video_path /data/new_dataset/video.mp4 \
    --mode windowed \
    --window_size 128 \
    --overlap_keyframes 8

```

Increase the **overlap** between windows to preserve continuity, and reduce optimization iterations to minimize accumulated drift:

```bash
python demo.py \
    --image_folder /data/new_dataset \
    --camera_num_iterations 1

```

### Dataset Format Mismatches

LingBot-Map expects sequentially-named images (e.g., `000001.png`) or video files accompanied by a [`metadata.json`](https://github.com/Robbyant/lingbot-map/blob/main/metadata.json) containing camera intrinsics. Raw datasets often trigger "cannot find image" or "invalid intrinsic matrix" errors.

The reference implementation in [`preprocess/oxford.py`](https://github.com/Robbyant/lingbot-map/blob/main/preprocess/oxford.py) demonstrates the required structure. Always preprocess raw data before inference:

```bash
python preprocess/oxford.py \
    --input_dir /raw/oxford_spires \
    --output_dir /processed/oxford_spires \
    --downsample 2

```

Verify the output contains zero-padded image indices and a valid [`metadata.json`](https://github.com/Robbyant/lingbot-map/blob/main/metadata.json) before running [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py).

## Step-by-Step Debugging Protocol

Follow this structured checklist when encountering failures on new data:

| Step | Verification | Command or Setting |
|------|--------------|-------------------|
| 1 | **FlashInfer Backend** | `pip show flashinfer-python` |
| 2 | **KV-Cache Occupancy** | `python gct_profile.py --backend flashinfer` |
| 3 | **Keyframe Interval** | `--keyframe_interval 2` |
| 4 | **Windowed Mode** | `--mode windowed --window_size 128` |
| 5 | **Window Overlap** | `--overlap_keyframes 8` |
| 6 | **Dataset Preprocessing** | `python preprocess/oxford.py --input_dir raw --output_dir processed` |
| 7 | **Trajectory Validation** | Open Visor at `http://localhost:8080` after running [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py) |
| 8 | **Memory Profiling** | Compare baselines with `gct_profile.py --backend flashinfer --dtype bf16 --compile` |

## Practical Code Examples for Debugging

### Diagnose Memory Pressure with FlashInfer Profiling

Launch inference with FlashInfer enabled, then monitor resource utilization in a separate terminal:

```bash

# Terminal 1: Start inference

python demo.py \
    --model_path /path/to/lingbot-map-long.pt \
    --image_folder /data/new_dataset \
    --mask_sky \
    --use_flashinfer

# Terminal 2: Monitor KV-cache

python gct_profile.py --backend flashinfer --dtype bf16 --compile

```

The profiler outputs real-time statistics:

```

KV cache slots: 120 / 128 (94%)
Peak memory: 7.2 GB

```

If occupancy exceeds 90%, abort and relaunch with `--keyframe_interval 4`.

### Run Windowed Inference on 10,000-Frame Sequences

For extremely long videos that exceed GPU memory constraints, use windowed mode with strategic overlap:

```bash
python demo.py \
    --video_path /data/new_dataset/video.mp4 \
    --model_path /path/to/lingbot-map-long.pt \
    --mode windowed \
    --window_size 128 \
    --overlap_keyframes 8 \
    --keyframe_interval 2 \
    --camera_num_iterations 2

```

This configuration resets the KV-cache every 128 frames while maintaining geometric continuity through 8 overlapping keyframes.

### Preprocess Oxford Spires-Compatible Data

Adapt custom datasets to the expected format using the Oxford preprocessor:

```bash
python preprocess/oxford.py \
    --input_dir /raw/custom_dataset \
    --output_dir /processed/custom_dataset \
    --downsample 2

```

Verify the generated structure:

```bash
ls /processed/custom_dataset | head -5

# Output: 000001.png  000002.png  000003.png  metadata.json

```

Inspect [`metadata.json`](https://github.com/Robbyant/lingbot-map/blob/main/metadata.json) to confirm camera intrinsics are present before running [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py).

### Mitigate Pose Collapse with Reduced Iterations

When experiencing trajectory drift, reduce the number of camera pose refinement iterations:

```bash
python demo.py \
    --image_folder /data/new_dataset \
    --model_path /path/to/lingbot-map-long.pt \
    --camera_num_iterations 1 \
    --mode windowed \
    --window_size 128

```

If drift persists, increase `--overlap_keyframes` or segment the sequence into shorter clips.

## Key Source Files for Deep Debugging

Understanding these specific implementation files accelerates root-cause analysis:

- **[`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py)**: Implements the paged KV-cache; inspect this when debugging OOM errors or memory spikes.
- **[`lingbot_map/layers/rope.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/rope.py)**: Contains the rotary positional embedding logic that causes pose-collapse beyond 320 frames.
- **[`gct_profile.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_profile.py)**: Utility script that prints KV-cache occupancy and peak GPU memory usage.
- **[`preprocess/oxford.py`](https://github.com/Robbyant/lingbot-map/blob/main/preprocess/oxford.py)**: Reference template for dataset formatting requirements.
- **[`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py)**: Primary entry point containing all CLI flags for inference configuration.
- **[`demo_render/batch_demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo_render/batch_demo.py)**: Alternative offline rendering pipeline more tolerant to very long sequences when [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py) exhausts memory.

## Summary

When running LingBot-Map on new datasets, failures typically concentrate in three areas:

- **Memory exhaustion** from unbounded KV-cache growth, resolvable via FlashInfer backend, reduced keyframe intervals, or windowed inference.
- **Pose drift** beyond the 320-frame training limit, mitigated by windowed mode with overlapping keyframes and reduced iteration counts.
- **Format mismatches** requiring strict preprocessing to match Oxford Spires conventions (zero-padded indices and [`metadata.json`](https://github.com/Robbyant/lingbot-map/blob/main/metadata.json)).

Use [`gct_profile.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_profile.py) to quantify memory pressure, and consult [`lingbot_map/layers/rope.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/rope.py) and [`lingbot_map/layers/flashinfer_cache.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/flashinfer_cache.py) when implementing custom fixes.

## Frequently Asked Questions

### What is the maximum sequence length LingBot-Map can handle without windowing?

The model was trained on sequences up to approximately **320 frames**. Beyond this limit, the RoPE embeddings in [`lingbot_map/layers/rope.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/rope.py) become unstable and the KV-cache overflows, necessitating **windowed mode** (`--mode windowed`) for longer videos.

### Why does my trajectory suddenly jump after frame 320?

This is **pose-collapse** caused by the KV-cache exceeding the model's training distribution. When the cache contains too many frames, the relative pose embeddings degrade. Enable windowed inference with `--window_size 128` and `--overlap_keyframes 8` to reset the cache periodically while preserving continuity.

### How do I verify that FlashInfer is installed and active?

Run `pip show flashinfer-python` to confirm installation. During inference, check that `--use_flashinfer` is passed to [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py) or ensure it is the default backend. Verify active backend selection by running `python gct_profile.py --backend flashinfer`, which will fail if FlashInfer is unavailable.

### What preprocessing is required for custom datasets versus the Oxford Spires benchmark?

All datasets must provide sequential images with zero-padded filenames (e.g., `000001.png`) and a [`metadata.json`](https://github.com/Robbyant/lingbot-map/blob/main/metadata.json) file containing camera intrinsics. Use `python preprocess/oxford.py` as a template pipeline, adapting the `--input_dir` and `--output_dir` paths to your custom data. The preprocessor handles downsampling and JSON generation required by [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py).