# How LingBot-Map's Trajectory Memory Mechanism Corrects Long-Range Drift

> LingBot-Map's trajectory memory mechanism uses a NumPy cache for global alignment, correcting long-range drift in one step to prevent error propagation for improved accuracy.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: how-to-guide
- Published: 2026-06-29

---

**LingBot-Map stores camera-to-world poses in a persistent NumPy cache that enables global alignment of the entire trajectory, correcting accumulated drift in a single operation rather than propagating errors frame-by-frame.**

LingBot-Map, an open-source visual odometry system from the Robbyant/lingbot-map repository, processes arbitrarily long video sequences while maintaining constant GPU memory usage. Its **trajectory memory mechanism** persists camera-to-world (C2W) poses to disk and performs global alignment on the cached trajectory to eliminate long-range drift. This architecture separates short-term inference windows from long-term pose history, enabling accurate correction across thousands of frames.

## Persistent Pose Caching Architecture

At the core of the system lies the `SceneCache` class defined in [`benchmark/viewer.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/viewer.py) (lines [83-110](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/viewer.py#L83-L110)). Instead of retaining all poses in GPU memory, the system writes them to a compact **NumPy `.npz` cache** on disk.

The `SceneCache` handles I/O operations through utilities in [`benchmark/benchmark/io/trajectory.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/io/trajectory.py), which provide `read_trajectory` and `write_trajectory` functions. These methods serialize C2W matrices and associated point clouds, enabling the system to process sequences of unlimited length without exhausting video memory.

### Separation of Concerns

The trajectory memory creates a strict separation between:

- **Active inference state**: Only the current sliding window of frames resides in GPU memory
- **Historical trajectory**: Older poses remain on disk until explicitly loaded for global operations

This design pattern appears in the streaming model implementations (`lingbot_map/models/gct_stream_window*.py`), where the KV-cache size remains **O(window)** rather than growing with sequence length.

## Global Alignment for Drift Correction

The trajectory memory enables **long-range drift correction** through a post-hoc global alignment step. After processing a sequence, the cached trajectory undergoes a similarity transformation to align with ground-truth data.

### Umeyama Similarity Transform

The alignment logic resides in [`benchmark/benchmark/evaluation/trajectory.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/evaluation/trajectory.py), which implements the `compute_umeyama` function. This method calculates the optimal similarity transform (rotation, translation, and scale) that minimizes the distance between the estimated trajectory and the reference trajectory.

In `SceneCache.load` (lines [52-66](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/viewer.py#L52-L66)), the system checks the `is_aligned` flag. If the trajectory requires correction, it applies the Umeyama transform to the entire pose history at once, removing accumulated drift from thousands of frames in a single operation.

### Constant-Time Pose Lookup

Because poses are stored on disk and loaded on demand, the system achieves **constant-time lookup** for any historical frame. The trajectory memory acts as a random-access store where the sliding-window models can retrieve historic poses via `traj_memory.get_poses(up_to=current_frame_idx)` without reloading the entire sequence into GPU memory.

## Integration with Streaming Inference

The trajectory memory mechanism integrates seamlessly with LingBot-Map's sliding-window attention models. In `lingbot_map/models/gct_stream_window*.py`, the streaming inference loop maintains only the latest window of frames in GPU memory while accessing the full pose history through the trajectory cache.

### Memory-Bounded Attention

When processing a new window, the model combines cached historic poses with current window estimates:

```python

# Inside a streaming GCT model (e.g., gct_stream_window_v2.py)

def forward(self, frames):
    # frames: [B, S, C, H, W] where S is the sliding window length

    historic_poses = self.traj_memory.get_poses(up_to=self.current_frame_idx)
    combined = torch.cat([historic_poses, self.window_poses], dim=1)
    # Proceed with attention using the combined pose sequence

    ...

```

This approach ensures that **KV-cache size scales with window size, not sequence length**, while still enabling accurate pose updates that account for the full trajectory history.

## Implementation Examples

The following examples demonstrate working with the trajectory memory system.

### Loading and Aligning Cached Trajectories

To load a scene's cached trajectory and apply global alignment:

```python
from benchmark.viewer import SceneCache
from benchmark.benchmark.evaluation.trajectory import compute_umeyama
from pathlib import Path

workspace_root = Path("/path/to/workspace")
data_dir = workspace_root / "my_dataset" / "scene01" / "lingbot_map"

# Initialize cache with spatial subsampling

cache = SceneCache(
    data_dir,
    spatial_subsample=8,
    workspace_root=workspace_root,
    remove_sky=False
)

# Load cached poses and point clouds

traj_dict, point_clouds, is_aligned = cache.load()

# Apply Umeyama transform if not already aligned

if not is_aligned:
    aligned_traj = compute_umeyama(traj_dict)
    cache.save(
        aligned_traj,
        point_clouds,
        num_frames=len(aligned_traj),
        image_width=640,
        image_height=480,
        is_aligned=True
    )

```

### Special Token Expansion

The base aggregator in [`lingbot_map/aggregator/base.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/base.py) (lines [30-58](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/base.py#L30-L58)) demonstrates how pose information flows into the trajectory memory. Special tokens containing pose data are expanded and flattened before feeding into the caching system, ensuring consistent C2W matrix formatting across the pipeline.

## Summary

- **Persistent NumPy caching**: The `SceneCache` class in [`benchmark/viewer.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/viewer.py) stores C2W poses in `.npz` files, enabling processing of unlimited sequence lengths without GPU memory exhaustion.
- **Single-operation drift correction**: The `compute_umeyama` function applies a global similarity transform to the entire cached trajectory, correcting accumulated drift across thousands of frames simultaneously.
- **Constant memory inference**: Sliding-window models in `gct_stream_window*.py` maintain O(window) GPU memory complexity by offloading historical poses to the trajectory cache.
- **Deterministic replay**: Cached trajectories support reproducible visualization and downstream point-cloud generation without re-running inference.

## Frequently Asked Questions

### What is the trajectory memory mechanism in LingBot-Map?

The trajectory memory mechanism is a persistent storage system that caches camera-to-world poses to disk using NumPy `.npz` files. Implemented via the `SceneCache` class in [`benchmark/viewer.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/viewer.py), it separates long-term pose history from active GPU memory, allowing the system to reference arbitrary past frames without reloading the entire sequence into video memory.

### How does global alignment correct long-range drift?

Global alignment corrects drift by applying an **Umeyama similarity transform** to the entire cached trajectory at once, as implemented in [`benchmark/benchmark/evaluation/trajectory.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/evaluation/trajectory.py). This single mathematical operation accounts for accumulated rotation, translation, and scale errors across all frames simultaneously, rather than propagating corrections sequentially through the windowed inference process.

### Why does LingBot-Map use NumPy `.npz` instead of GPU memory for poses?

Storing poses in NumPy `.npz` files via the `SceneCache` allows the system to maintain **constant GPU memory usage** regardless of video length. The sliding-window attention models only keep the current inference window in GPU memory (O(window) complexity), while the trajectory memory provides on-demand access to historical poses, preventing out-of-memory errors during long-sequence processing.

### Which model files implement the trajectory memory integration?

The sliding-window GCT streamers in `lingbot_map/models/gct_stream_window*.py` integrate with the trajectory memory to bound memory usage. These models access historic poses through `traj_memory.get_poses()` while keeping only the active window in GPU tensors, as coordinated with the `SceneCache` I/O utilities in [`benchmark/benchmark/io/trajectory.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/io/trajectory.py).