How 3D RoPE Enables Temporal Consistency in LingBot-Map

3D RoPE in LingBot-Map assigns global frame indices to rotary position embeddings, ensuring that cached keys and queries maintain temporal alignment across streaming video frames by rotating features once before they enter the KV cache.

LingBot-Map is an open-source streaming vision transformer designed for SLAM and map-building tasks. By extending traditional 2D Rotary Position Embeddings (RoPE) to three dimensions, the repository solves the critical challenge of maintaining temporal consistency during incremental frame processing, where the model must align new observations with cached historical states.

What is 3D RoPE?

Rotary Position Embedding (RoPE) encodes token positions by rotating feature vectors using sinusoidal matrices. While standard 2D RoPE handles spatial dimensions independently, 3D RoPE introduces an additional temporal dimension to manage frame sequences in video streams.

From 2D to 3D Rotary Position Embeddings

Standard 2D RoPE, as implemented in lingbot_map/layers/rope.py, treats the height and width of a patch grid as independent axes and concatenates their rotated components. 3D RoPE extends this paradigm to include a temporal dimension representing the frame index, allowing the model to distinguish between tokens from different time steps while preserving geometric relationships within each frame.

Implementation in WanRotaryPosEmbed

The core implementation resides in the WanRotaryPosEmbed class within lingbot_map/layers/rope.py. The constructor receives the attention head dimension and patch size, then splits the head dimension into three sub-dimensions for time, height, and width:

# lingbot_map/layers/rope.py

class WanRotaryPosEmbed(nn.Module):
    """3D rotary position embedding (t, h, w)."""
    def __init__(self, attention_head_dim, patch_size, max_seq_len=1024,
                 theta=10000.0, fhw_dim=[20, 22, 22]):
        # Split head_dim into t-, h-, w-sub-dimensions

        # Pre-compute separate 1-D RoPE tables for each axis (get_1d_rotary_pos_embed)

        # Store them in self.freqs (complex-valued tensor)

The fhw_dim parameter determines the allocation of head dimensions across temporal, height, and width axes, ensuring that t_dim + h_dim + w_dim = head_dim. The system pre-computes frequency tables for each axis using get_1d_rotary_pos_embed, concatenating them into self.freqs for efficient indexing with frame-row-column triplets.

How 3D RoPE Provides Temporal Consistency

Temporal consistency in streaming transformers requires that keys and values stored in the KV cache use identical global frame indices, regardless of whether the model processes single frames or sliding windows.

The Global Frame Index Mechanism

Unlike relative position encoding that resets with each window, 3D RoPE uses absolute global frame indices. When the camera head generates position embeddings, it passes the global start and end frame indices to the RoPE module:

# lingbot_map/heads/camera_head.py (excerpt)

if self.rope3d is not None:
    pos3d = self.rope3d(
        ppf=S,               # Number of frames in current chunk

        pph=1, ppw=1,        # Camera token occupies 1×1 patch

        patch_start_idx=0,
        device=pose_tokens.device,
        f_start=self.frame_idx,         # Global start frame

        f_end=self.frame_idx + S,       # Global end frame

    )

This ensures that frame 100 receives the same positional encoding whether processed in isolation or as part of a sliding window starting at frame 98, eliminating the drift that occurs with relative indexing.

Integration with the KV Cache

The WanRotaryPosEmbed.__call__ method splits pre-computed frequencies into temporal, height, and width components, selecting the slice belonging to the current frame range. It returns a complex-valued tensor of shape [1, 1, S, head_dim//2], which is supplied to the attention block via the pos=pos3d parameter before keys enter the KV cache.

Consequently, when the KV cache is later read (as in lingbot_map/layers/attention.py), the stored keys already contain the correct temporal rotation, ensuring all cached frames align on the same global timeline.

Preventing Double Application

The attention mechanism guards against re-applying positional encoding to cached keys. According to lines 154-159 of lingbot_map/layers/attention.py, the system checks if enable_3d_rope before applying standard 2D RoPE:

# Conditional logic in attention block

if enable_3d_rope:
    # Use pre-computed 3D positions

    pass
else:
    # Apply standard 2D RoPE

This guarantees that temporal rotation is applied exactly once and never re-applied when retrieving cached keys, preserving the phase alignment critical for temporal consistency.

Implementation Flow in LingBot-Map

The 3D RoPE system propagates through the model architecture via a cascading configuration flag, ensuring consistent behavior across all components.

Model Configuration

The enable_3d_rope flag originates in GCTBase (line 56 of lingbot_map/models/gct_base.py) and flows through GCTStream to the aggregators and heads. Users enable temporal consistency by setting this boolean flag during model instantiation:

from lingbot_map.models.gct_stream_window_v2 import GCTStream

model = GCTStream(
    img_size=518,
    patch_size=14,
    embed_dim=1024,
    enable_3d_rope=True,  # Enable temporal consistency

    max_frame_num=1024,
)

Aggregator Initialization

When AggregatorStream receives enable_3d_rope=True, it initializes the 3D RoPE module via _init_3d_rope() (lines 101-104 of lingbot_map/aggregator/stream.py). This creates a shared WanRotaryPosEmbed instance used by all transformer blocks in the streaming pipeline, ensuring uniform position encoding across the architecture.

Camera Head Position Generation

In CameraCausalHead.__init__ (lines 86-88 of lingbot_map/heads/camera_head.py), the 3D RoPE module is instantiated conditionally when the flag is enabled. During each forward pass, the head generates frame-wise positional tensors using the global frame counter (self.frame_idx), ensuring that each new token receives the correct temporal rotation based on its absolute position in the video sequence.

Practical Usage Example

The following example demonstrates enabling 3D RoPE for streaming inference with the GCTStream model:

import torch
from lingbot_map.models.gct_stream_window_v2 import GCTStream
from lingbot_map.utils.geometry import setup_dummy_input

# Configure model with 3D RoPE enabled

model = GCTStream(
    img_size=518,
    patch_size=14,
    embed_dim=1024,
    disable_global_rope=False,
    enable_3d_rope=True,  # Critical for temporal consistency

    max_frame_num=1024,
)

# Simulate streaming batch (B=1, 3 frames, 1 camera token per frame)

tokens, mask = setup_dummy_input(num_frames=3)

# Forward pass with causal inference

with torch.no_grad():
    out = model(tokens, mask=mask, causal_inference=True)

print(out.shape)  # Output: (B, S, output_dim) with temporally aligned embeddings

The setup_dummy_input helper creates tensors shaped [B, S, C] where S represents the number of frames, mimicking the token embeddings produced by the vision backbone in production systems.

Summary

• 3D RoPE extends 2D rotary embeddings with a temporal dimension through the WanRotaryPosEmbed class in lingbot_map/layers/rope.py, splitting attention head dimensions across time, height, and width axes.
• Global frame indices ensure that identical frames receive identical positional encodings regardless of sliding window position, preventing temporal drift during streaming inference.
• Pre-caching rotation applies temporal encoding via f_start and f_end parameters before keys enter the KV cache, maintaining alignment across variable-length sequences.
• Flag-based initialization propagates via enable_3d_rope from GCTBase through AggregatorStream to CameraCausalHead, coordinating the entire pipeline while preventing double-application in the attention blocks.

Frequently Asked Questions

What is the difference between 2D and 3D RoPE?

2D RoPE treats spatial dimensions (height and width) independently, concatenating their rotated components. 3D RoPE adds a temporal dimension (frame index) to the rotation, splitting the attention head dimension across time, height, and width axes. In LingBot-Map, this allows the transformer to distinguish between frames in a video stream while maintaining spatial relationships within each frame.

Why does temporal consistency require global frame indices?

Relative frame indices reset with each sliding window, causing the same absolute frame to receive different positional encodings depending on its position within the window. By using global frame indices (via f_start and f_end in CameraCausalHead), 3D RoPE ensures that frame 100 always rotates by the same phase angle, whether processed as part of a window starting at frame 98 or frame 95, eliminating temporal drift.

How does 3D RoPE prevent drift in streaming inference?

Drift occurs when cached keys and new queries use different positional reference frames. 3D RoPE prevents this by applying temporal rotation once before keys enter the KV cache (in lingbot_map/layers/attention.py), using absolute global indices. When the cache is later read, the stored keys already contain the correct temporal phase, ensuring alignment with newly computed queries. The attention block also guards against double-application via the enable_3d_rope flag.

Where is the 3D RoPE module instantiated in LingBot-Map?

The module is instantiated in AggregatorStream._init_3d_rope() (lines 101-104 of lingbot_map/aggregator/stream.py) and passed to CameraCausalHead during initialization (lines 86-88 of lingbot_map/heads/camera_head.py). The WanRotaryPosEmbed class itself is defined in lingbot_map/layers/rope.py, and the enabling flag originates in GCTBase (line 56 of lingbot_map/models/gct_base.py).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}