# Trajectory Memory Mechanism in the Geometric Context Transformer: Architecture and Implementation

> Explore the trajectory memory mechanism in the Geometric Context Transformer. Learn how it uses a sliding-window cache for causal attention, drift correction, and efficient GPU memory management.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: architecture
- Published: 2026-07-04

---

**The trajectory memory mechanism in the Geometric Context Transformer maintains a sliding-window key-value cache that stores pose encodings from past frames, enabling causal attention and drift correction while bounding GPU memory usage to O(window) complexity regardless of video length.**

The Geometric Context Transformer (GCT) in the `Robbyant/lingbot-map` repository addresses the challenge of processing arbitrarily long video streams by implementing a trajectory memory mechanism that preserves temporal consistency without requiring unbounded computational resources. This system unifies dense geometric reasoning with long-range temporal modeling by maintaining a compact, evictable cache of past camera poses and visual tokens inside the transformer blocks.

## How the Trajectory Memory Works

At its core, the trajectory memory is realized through a **key-value (KV) cache** embedded within the transformer blocks of the streaming aggregator and causal camera head. When processing a frame, each transformer block stores **key** and **value** tensors encoding the visual tokens, creating a compact representation that can be reused for later attention calculations without recomputing the entire sequence.

This cache lives in [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py) and is managed through methods such as `clean_kv_cache` and `get_kv_cache_info`, with internal attributes prefixed by `kv_cache_*` initialized in the model constructor.

## Sliding-Window Cache Management

To prevent memory explosion during long videos, GCT implements a strict **sliding-window eviction policy**. The cache is bounded by the configurable parameter `kv_cache_sliding_window`, which ensures that only the most recent *N* frames remain in GPU memory.

According to the source code in [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py) (lines 126-133), the model tracks `kv_cache_sliding_window` and `kv_cache_scale_frames` to manage memory allocation. When the cache exceeds the window limit, the system triggers eviction logic via `_set_defer_eviction` and `_execute_deferred_eviction`, guaranteeing O(window) memory usage even for arbitrarily long input sequences.

## Keyframe-Based Memory Optimization

Not every frame permanently enters the trajectory memory. GCT employs a **keyframe detection strategy** controlled by `keyframe_interval` or flow-based heuristics to determine whether a frame's KV pair should persist in the cache.

Inside `inference_streaming` (lines 106-114), the logic evaluates `_set_skip_append` to decide cache insertion. Non-keyframes are processed using the cached context but immediately discarded via `_rollback_last_frame`, reducing cache growth by a factor equal to the keyframe interval while maintaining access to recent temporal context.

## Trajectory-Memory Query and Pose Encoding

The trajectory memory serves as a persistent reference for camera pose estimation. The model extracts **pose encodings** (`pose_enc`) from the cached KV tensors at any timestep, which encode the camera center, rotation (as a quaternion), and intrinsics.

In [`lingbot_map/models/gct_base.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_base.py), the `_predict_camera` method returns the `"pose_enc"` tensor, which is subsequently converted to extrinsic and intrinsic matrices via `pose_encoding_to_extri_intri`. These encodings feed into the `CameraCausalHead` for iterative pose refinement and depth-to-world projection, enabling the model to correct drift by anchoring current predictions to the remembered trajectory.

## Cross-Window Alignment for Global Consistency

When processing videos in overlapping windows, GCT ensures trajectory consistency through **cross-window alignment**. Each window's pose memory first aligns to the previous window using a similarity transform (scale *s*, rotation *R*, translation *t*) computed from the overlapping keyframes' stored trajectory encodings.

The implementation in [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py) provides `_pairwise_alignment` and `_warp_predictions` methods, orchestrated by the high-level `inference_windowed` API. This mechanism eliminates drift accumulation that would otherwise occur between independent window inferences.

## Implementing Trajectory Memory in Practice

The following example demonstrates how to initialize a streaming GCT model with trajectory memory configuration and inspect cache statistics:

```python
import torch
from lingbot_map.models.gct_stream import GCTStream

# Initialize streaming GCT with trajectory memory constraints

model = GCTStream(
    sliding_window_size=64,          # Retain 64 frames in KV cache

    keyframe_interval=4,             # Store every 4th frame permanently

    enable_3d_rope=True,             # Enable temporal RoPE for consistency

)

# Simulate video input: [Batch, Sequence, Channels, Height, Width]

video = torch.randn(1, 120, 3, 224, 224)

# Run streaming inference with automatic keyframe detection

outputs = model.inference_streaming(
    video,
    keyframe_interval=4,
    flow_threshold=0.0,              # Disable flow-based keyframe detection

)

# Inspect trajectory memory statistics

memory_info = model.get_kv_cache_info()
print(f"Cached blocks: {memory_info['num_cached_blocks']}, "
      f"Memory (MiB): {memory_info['cache_memory_mb']:.2f}")

# Clear trajectory memory before processing a new video

model.clean_kv_cache()

```

The `get_kv_cache_info()` method returns the number of cached transformer blocks and an estimate of GPU memory consumed by the trajectory memory, while `clean_kv_cache()` resets the state for new sequences.

## Summary

- **The trajectory memory mechanism** in the Geometric Context Transformer uses a sliding-window KV cache to maintain temporal consistency with bounded memory usage.
- **Cache management** operates through configurable sliding windows (`kv_cache_sliding_window`) and keyframe selection (`keyframe_interval`) to optimize GPU memory in [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py).
- **Pose encodings** stored in the cache enable drift correction via `pose_encoding_to_extri_intri` and support iterative refinement in the camera head.
- **Cross-window alignment** methods (`_pairwise_alignment`, `_warp_predictions`) ensure global trajectory consistency when processing long videos in segments.
- **Public APIs** `inference_streaming` and `inference_windowed` provide convenient access to the trajectory memory functionality with automatic cache management.

## Frequently Asked Questions

### How does the trajectory memory mechanism prevent GPU memory overflow during long video processing?

The trajectory memory mechanism enforces a **sliding-window eviction policy** via the `kv_cache_sliding_window` parameter in [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py). Once the cache exceeds the configured window size, the `_execute_deferred_eviction` method removes the oldest entries, ensuring memory usage scales as O(window) rather than O(sequence length), allowing processing of arbitrarily long videos on single-GPU systems.

### What is the difference between keyframes and regular frames in the GCT cache?

**Keyframes** are selected frames (controlled by `keyframe_interval` or flow thresholds) whose key-value tensors persist in the trajectory memory, while regular frames are processed using the cached context but immediately discarded via `_rollback_last_frame`. This distinction reduces cache growth by only retaining geometrically significant frames, trading storage efficiency against temporal context density.

### How does cross-window alignment maintain global trajectory consistency?

Cross-window alignment uses stored pose encodings from overlapping frames between consecutive windows to compute a similarity transform (scale, rotation, and translation) via `_pairwise_alignment` in [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py). The `_warp_predictions` method applies this transform to align new predictions with the existing trajectory memory, preventing drift accumulation that would otherwise occur when processing video segments independently.

### Where is the pose encoding stored in the Geometric Context Transformer architecture?

Pose encodings are stored as **value tensors** within the KV cache of the transformer blocks, specifically managed by the `AggregatorStream` module and consumed by the `CameraCausalHead`. The `_predict_camera` method in [`lingbot_map/models/gct_base.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_base.py) extracts these encodings (`pose_enc`), which are subsequently decoded by `pose_encoding_to_extri_intri` in [`lingbot_map/utils/pose_enc.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/pose_enc.py) to recover camera extrinsics and intrinsics.