Trajectory Memory Mechanism in the Geometric Context Transformer: Architecture and Implementation

The trajectory memory mechanism in the Geometric Context Transformer maintains a sliding-window key-value cache that stores pose encodings from past frames, enabling causal attention and drift correction while bounding GPU memory usage to O(window) complexity regardless of video length.

The Geometric Context Transformer (GCT) in the Robbyant/lingbot-map repository addresses the challenge of processing arbitrarily long video streams by implementing a trajectory memory mechanism that preserves temporal consistency without requiring unbounded computational resources. This system unifies dense geometric reasoning with long-range temporal modeling by maintaining a compact, evictable cache of past camera poses and visual tokens inside the transformer blocks.

How the Trajectory Memory Works

At its core, the trajectory memory is realized through a key-value (KV) cache embedded within the transformer blocks of the streaming aggregator and causal camera head. When processing a frame, each transformer block stores key and value tensors encoding the visual tokens, creating a compact representation that can be reused for later attention calculations without recomputing the entire sequence.

This cache lives in lingbot_map/models/gct_stream.py and is managed through methods such as clean_kv_cache and get_kv_cache_info, with internal attributes prefixed by kv_cache_* initialized in the model constructor.

Sliding-Window Cache Management

To prevent memory explosion during long videos, GCT implements a strict sliding-window eviction policy. The cache is bounded by the configurable parameter kv_cache_sliding_window, which ensures that only the most recent N frames remain in GPU memory.

According to the source code in lingbot_map/models/gct_stream.py (lines 126-133), the model tracks kv_cache_sliding_window and kv_cache_scale_frames to manage memory allocation. When the cache exceeds the window limit, the system triggers eviction logic via _set_defer_eviction and _execute_deferred_eviction, guaranteeing O(window) memory usage even for arbitrarily long input sequences.

Keyframe-Based Memory Optimization

Not every frame permanently enters the trajectory memory. GCT employs a keyframe detection strategy controlled by keyframe_interval or flow-based heuristics to determine whether a frame's KV pair should persist in the cache.

Inside inference_streaming (lines 106-114), the logic evaluates _set_skip_append to decide cache insertion. Non-keyframes are processed using the cached context but immediately discarded via _rollback_last_frame, reducing cache growth by a factor equal to the keyframe interval while maintaining access to recent temporal context.

Trajectory-Memory Query and Pose Encoding

The trajectory memory serves as a persistent reference for camera pose estimation. The model extracts pose encodings (pose_enc) from the cached KV tensors at any timestep, which encode the camera center, rotation (as a quaternion), and intrinsics.

In lingbot_map/models/gct_base.py, the _predict_camera method returns the "pose_enc" tensor, which is subsequently converted to extrinsic and intrinsic matrices via pose_encoding_to_extri_intri. These encodings feed into the CameraCausalHead for iterative pose refinement and depth-to-world projection, enabling the model to correct drift by anchoring current predictions to the remembered trajectory.

Cross-Window Alignment for Global Consistency

When processing videos in overlapping windows, GCT ensures trajectory consistency through cross-window alignment. Each window's pose memory first aligns to the previous window using a similarity transform (scale s, rotation R, translation t) computed from the overlapping keyframes' stored trajectory encodings.

The implementation in lingbot_map/models/gct_stream.py provides _pairwise_alignment and _warp_predictions methods, orchestrated by the high-level inference_windowed API. This mechanism eliminates drift accumulation that would otherwise occur between independent window inferences.

Implementing Trajectory Memory in Practice

The following example demonstrates how to initialize a streaming GCT model with trajectory memory configuration and inspect cache statistics:

import torch
from lingbot_map.models.gct_stream import GCTStream

# Initialize streaming GCT with trajectory memory constraints

model = GCTStream(
    sliding_window_size=64,          # Retain 64 frames in KV cache

    keyframe_interval=4,             # Store every 4th frame permanently

    enable_3d_rope=True,             # Enable temporal RoPE for consistency

)

# Simulate video input: [Batch, Sequence, Channels, Height, Width]

video = torch.randn(1, 120, 3, 224, 224)

# Run streaming inference with automatic keyframe detection

outputs = model.inference_streaming(
    video,
    keyframe_interval=4,
    flow_threshold=0.0,              # Disable flow-based keyframe detection

)

# Inspect trajectory memory statistics

memory_info = model.get_kv_cache_info()
print(f"Cached blocks: {memory_info['num_cached_blocks']}, "
      f"Memory (MiB): {memory_info['cache_memory_mb']:.2f}")

# Clear trajectory memory before processing a new video

model.clean_kv_cache()

The get_kv_cache_info() method returns the number of cached transformer blocks and an estimate of GPU memory consumed by the trajectory memory, while clean_kv_cache() resets the state for new sequences.

Summary

• The trajectory memory mechanism in the Geometric Context Transformer uses a sliding-window KV cache to maintain temporal consistency with bounded memory usage.
• Cache management operates through configurable sliding windows (kv_cache_sliding_window) and keyframe selection (keyframe_interval) to optimize GPU memory in lingbot_map/models/gct_stream.py.
• Pose encodings stored in the cache enable drift correction via pose_encoding_to_extri_intri and support iterative refinement in the camera head.
• Cross-window alignment methods (_pairwise_alignment, _warp_predictions) ensure global trajectory consistency when processing long videos in segments.
• Public APIs inference_streaming and inference_windowed provide convenient access to the trajectory memory functionality with automatic cache management.

Frequently Asked Questions

How does the trajectory memory mechanism prevent GPU memory overflow during long video processing?

The trajectory memory mechanism enforces a sliding-window eviction policy via the kv_cache_sliding_window parameter in lingbot_map/models/gct_stream.py. Once the cache exceeds the configured window size, the _execute_deferred_eviction method removes the oldest entries, ensuring memory usage scales as O(window) rather than O(sequence length), allowing processing of arbitrarily long videos on single-GPU systems.

What is the difference between keyframes and regular frames in the GCT cache?

Keyframes are selected frames (controlled by keyframe_interval or flow thresholds) whose key-value tensors persist in the trajectory memory, while regular frames are processed using the cached context but immediately discarded via _rollback_last_frame. This distinction reduces cache growth by only retaining geometrically significant frames, trading storage efficiency against temporal context density.

How does cross-window alignment maintain global trajectory consistency?

Cross-window alignment uses stored pose encodings from overlapping frames between consecutive windows to compute a similarity transform (scale, rotation, and translation) via _pairwise_alignment in lingbot_map/models/gct_stream.py. The _warp_predictions method applies this transform to align new predictions with the existing trajectory memory, preventing drift accumulation that would otherwise occur when processing video segments independently.

Where is the pose encoding stored in the Geometric Context Transformer architecture?

Pose encodings are stored as value tensors within the KV cache of the transformer blocks, specifically managed by the AggregatorStream module and consumed by the CameraCausalHead. The _predict_camera method in lingbot_map/models/gct_base.py extracts these encodings (pose_enc), which are subsequently decoded by pose_encoding_to_extri_intri in lingbot_map/utils/pose_enc.py to recover camera extrinsics and intrinsics.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}