# How Trajectory Memory Tackles Long-Range Drift in LingBot-Map's 3D Reconstruction

> Discover how trajectory memory in LingBot-Map's 3D reconstruction solves long-range drift. Learn how global pose-graph optimization and Umeyama alignment eliminate accumulated drift.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: deep-dive
- Published: 2026-07-08

---

**Trajectory memory** stores every camera-to-world (C2W) pose in a persistent CPU buffer while the system processes frames in fixed-size GPU windows, enabling global pose-graph optimization and Umeyama alignment to eliminate accumulated drift across thousands of frames.

LingBot-Map, an open-source neural 3D reconstruction system available in the `Robbyant/lingbot-map` repository, solves the critical problem of long-range drift through a dedicated trajectory memory mechanism. Traditional streaming architectures discard historical poses to conserve GPU memory, but LingBot-Map maintains an unbounded CPU-side record of the entire camera path. This persistent storage allows the system to perform global trajectory alignment and loop-closure correction, ensuring reconstruction accuracy remains stable even on extended sequences.

## Sliding-Window KV-Cache with Pose Retention

The streaming transformer in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py) processes video frames within a fixed-size window to bound GPU memory usage, while the base aggregator in [`lingbot_map/aggregator/base.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/base.py) frees old feature checkpoints. However, the system decouples pose estimation from the attention cache by writing every predicted **camera-to-world (C2W) pose** into a long-term CPU buffer.

This architecture uses `self._trajectory_memory`, a CPU-side list that grows unbounded, to store poses indefinitely while the GPU-bound KV-cache cycles through new windows. After the model predicts the current pose `pose_c2w`, the system appends it to persistent storage:

```python

# After the model predicts the current pose `pose_c2w`

self._trajectory_memory.append(pose_c2w)   # CPU-side list that grows unbounded

```

By retaining every pose on the host while freeing GPU memory, LingBot-Map preserves the complete camera trajectory without sacrificing streaming performance.

## Incremental Pose Graph Optimization

Once poses enter the trajectory memory, LingBot-Map treats them as nodes in a global pose graph. When the system detects loop-closure cues—such as revisiting a previously mapped area—it triggers a global optimization that adjusts all stored poses simultaneously.

This incremental bundle adjustment pulls the entire trajectory back into alignment, distributing correction errors across the full history rather than accumulating unbounded drift at the tail end of the sequence. The optimization operates on the persistent pose records maintained outside the sliding window, enabling corrections that span thousands of frames.

## Global Map Alignment via Umeyama Transform

Before visualization or evaluation, LingBot-Map applies a **global similarity transform** to align the stored trajectory against a reference frame. This step corrects scale, rotation, and translation drift that accumulates over long sequences.

According to the source code in [`lingbot_map/utils/geometry.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/geometry.py), the system uses the `umeyama_transform` function to compute the optimal alignment between the current trajectory and a reference (such as the first keyframe or ground-truth data):

```python
from lingbot_map.utils.geometry import umeyama_transform

aligned_traj = umeyama_transform(old_traj, ref_traj)   # Apply to all stored poses

```

This transformation ensures that the visualized camera path remains metrically consistent with the global map coordinate system, neutralizing cumulative drift errors.

## Persistent Trajectory I/O in BSS v2 Format

The benchmark utilities in [`benchmark/benchmark/io/trajectory.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/io/trajectory.py) define a dedicated "BSS v2" format that stores every timestamped pose with full precision. This persistence layer enables downstream tools to reconstruct the exact camera path and quantify drift without information loss.

Loading a stored trajectory for analysis or visualization requires a single call:

```python
from benchmark.benchmark.io import trajectory as traj_io

# Load a trajectory saved by the benchmark (BSS v2 format)

traj = traj_io.read_trajectory(Path("outputs/scene_01/trajectory.bss"))

# `traj` is an (N, 13) array: (timestamp, tx, ty, tz, qw, qx, qy, qz)

```

The format stores 13 values per frame: timestamp, translation (tx, ty, tz), and quaternion rotation (qw, qx, qy, qz). This comprehensive encoding preserves the full SE(3) pose information required for accurate drift evaluation.

## Visualizing the Corrected Trajectory

The viewer implementation in [`benchmark/viewer.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/viewer.py) renders the aligned trajectory as a colored tube to verify global consistency. When visualization is enabled, the system constructs a mesh representation of the camera path:

```python
if self.glb_trajectory_checkbox.value and len(cam_positions) >= 2:
    traj_mesh = self._build_trajectory_tube(
        positions=cam_positions,
        radius=self.glb_trajectory_radius_slider.value,
        colormap="rainbow",
        num_cameras=len(cam_positions),
    )
    self.scene.add(traj_mesh, name="trajectory")

```

This visualization step confirms that the trajectory memory and alignment procedures have successfully eliminated long-range drift, producing a geometrically consistent camera path.

## Summary

- **Decoupled storage**: The system separates GPU-bound attention windows from CPU-bound pose persistence in [`gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream_window_v2.py) and [`aggregator/base.py`](https://github.com/Robbyant/lingbot-map/blob/main/aggregator/base.py), allowing indefinite trajectory retention without memory exhaustion.
- **Global optimization**: Trajectory memory enables pose-graph optimization that distributes loop-closure corrections across the entire camera path rather than accumulating local errors.
- **Metric alignment**: The `umeyama_transform` utility in [`lingbot_map/utils/geometry.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/geometry.py) applies global similarity transforms to correct scale and rotation drift before visualization.
- **Persistent format**: BSS v2 trajectory files in [`benchmark/benchmark/io/trajectory.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/io/trajectory.py) store complete SE(3) poses with timestamps, ensuring no data loss between processing and evaluation.
- **Validation tools**: The viewer in [`benchmark/viewer.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/viewer.py) renders aligned trajectories as colored tubes to verify drift correction visually.

## Frequently Asked Questions

### What is trajectory memory in LingBot-Map?

**Trajectory memory** is a persistent CPU-side buffer that stores every estimated camera-to-world pose throughout the entire reconstruction sequence. Unlike the GPU-bound sliding window cache that discards old frames to save memory, this buffer in [`gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream_window_v2.py) maintains an unbounded list of poses (`self._trajectory_memory`) enabling global trajectory optimization and drift correction across thousands of frames.

### How does the sliding window architecture interact with trajectory memory?

The streaming transformer processes frames in fixed-size windows to keep GPU memory bounded, but immediately writes each estimated pose to the CPU-side trajectory memory. This **decoupled architecture** allows the system to forget old feature maps while remembering every camera position, bounding computational memory without sacrificing the global pose information needed for drift correction.

### What file format stores the persistent trajectory data?

LingBot-Map uses the **BSS v2 format** implemented in [`benchmark/benchmark/io/trajectory.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/io/trajectory.py). This format stores each pose as a 13-element vector containing timestamp, translation (tx, ty, tz), and quaternion rotation (qw, qx, qy, qz). The `read_trajectory` and write functions enable exact reconstruction of the camera path for evaluation and visualization.

### How does Umeyama alignment correct long-range drift?

The `umeyama_transform` function in [`lingbot_map/utils/geometry.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/geometry.py) computes the optimal similarity transform (scale, rotation, translation) that aligns the current trajectory to a reference pose. By applying this global transformation to all stored poses in the trajectory memory, the system compensates for accumulated scale drift and orientation errors that would otherwise grow unbounded over long sequences.