What Is 3D RoPE and How Does It Achieve Temporal Consistency in LingBot-Map?

3D RoPE extends standard 2‑D rotary position embeddings with a temporal dimension that encodes global frame indices, ensuring cached keys and queries remain aligned across streaming inference by rotating features once before they enter the KV cache.

Robbyant's LingBot-Map is an open-source SLAM system that processes video streams through transformer blocks. To prevent temporal drift during sliding-window inference, the implementation introduces 3D Rotary Position Embedding (RoPE), which augments spatial coordinates with absolute temporal positioning to stabilize representations across frames.

What Is 3D RoPE?

Standard Rotary Position Embedding (RoPE) encodes token positions by rotating feature vectors using sinusoidal matrices. While 2‑D RoPE treats height and width as independent axes, 3D RoPE introduces a third axis for time, allowing the model to distinguish temporal positions using absolute global indices rather than relative window offsets.

The WanRotaryPosEmbed Implementation

The core logic resides in lingbot_map/layers/rope.py within the WanRotaryPosEmbed class. This module splits the attention head dimension into three distinct sub-dimensions for time (t), height (h), and width (w).

# lingbot_map/layers/rope.py

class WanRotaryPosEmbed(nn.Module):
    """3D rotary position embedding (t, h, w)."""
    def __init__(self, attention_head_dim, patch_size, max_seq_len=1024,
                 theta=10000.0, fhw_dim=[20, 22, 22]):
        # head_dim is split into t-, h-, and w-sub-dimensions

        # Pre-computes separate 1-D RoPE tables for each axis via get_1d_rotary_pos_embed

        # Stores concatenated frequencies in self.freqs as complex-valued tensors

The fhw_dim parameter (default [20, 22, 22]) determines how many units of the head dimension are allocated to each axis, where t_dim + h_dim + w_dim = head_dim. For each axis, the constructor invokes get_1d_rotary_pos_embed to build sinusoidal frequency tables, then concatenates these into self.freqs—a tensor indexable by (frame, row, column) triples.

Dimensional Isolation

By partitioning the embedding space, temporal rotations remain orthogonal to spatial patterns. For a configuration with attention_head_dim=64, the split might reserve 20 dimensions for temporal encoding, 22 for height, and 22 for width. This isolation prevents temporal positional information from interfering with spatial geometric reasoning.

How 3D RoPE Achieves Temporal Consistency

Temporal consistency requires that identical frames receive identical embeddings regardless of when they are processed in a streaming pipeline. 3D RoPE accomplishes this through global indexing and strategic placement in the attention mechanism.

Global Frame Indexing

Instead of using relative positions within a sliding window, CameraCausalHead passes absolute frame indices to the RoPE module. The head maintains self.frame_idx as a persistent global counter across inference steps:

# lingbot_map/heads/camera_head.py

if self.rope3d is not None:
    pos3d = self.rope3d(
        ppf=S,                       # Number of frames in current chunk

        pph=1, ppw=1,                # Camera token occupies 1×1 patch

        patch_start_idx=0,
        device=pose_tokens.device,
        f_start=self.frame_idx,      # Global start frame (absolute)

        f_end=self.frame_idx + S,    # Global end frame (absolute)

    )

The WanRotaryPosEmbed.__call__ method selects the frequency slice corresponding to the range [f_start:f_end], returning a complex tensor of shape [1, 1, S, head_dim//2]. Because this encoding derives deterministically from the absolute frame number, the same physical frame processed in different windows receives an identical rotation.

KV Cache Integration

The attention mechanism applies 3D RoPE before keys enter the KV cache. In lingbot_map/layers/attention.py, the code checks the enable_3d_rope flag to prevent double-application during cache retrieval:

# lingbot_map/layers/attention.py (lines 154-159)

if not enable_3d_rope:
    # Apply standard 2-D RoPE only when 3D is disabled

    apply_rotary_emb(...)

# Keys are now temporally encoded and stored in the cache

This ensures that when the model retrieves cached keys during streaming, all temporal rotations have already been applied. The cached representations remain fixed to the global timeline, eliminating the drift that would occur if relative window positions were used.

Flag Propagation Chain

The enable_3d_rope boolean propagates through the model hierarchy to coordinate the feature:

• Model Base: Defined in GCTBase.__init__ (lingbot_map/models/gct_base.py, line 56)
• Aggregator: AggregatorStream._init_3d_rope() instantiates the shared WanRotaryPosEmbed instance (lingbot_map/aggregator/stream.py, lines 101-104)
• Detection Head: CameraCausalHead initializes self.rope3d when the flag is active (lingbot_map/heads/camera_head.py, lines 86-88)

When enabled, this chain ensures that all transformer blocks share the same 3D RoPE module and global frame counter.

Practical Usage Example

To enable temporal consistency in a streaming configuration:

from lingbot_map.models.gct_stream_window_v2 import GCTStream

# Initialize with 3D RoPE enabled

model = GCTStream(
    img_size=518,
    patch_size=14,
    embed_dim=1024,
    disable_global_rope=False,
    enable_3d_rope=True,        # Enable 3-D temporal RoPE

    max_frame_num=1024,
)

# Streaming inference maintains temporal consistency automatically

# The model internally tracks frame_idx and applies appropriate rotations

During forward passes, the CameraCausalHead automatically manages frame indexing and applies 3D rotations to camera tokens without requiring manual position management.

Summary

• 3D RoPE extends 2‑D rotary embeddings by adding a temporal dimension via the WanRotaryPosEmbed class in lingbot_map/layers/rope.py.
• Dimensional splitting uses the fhw_dim parameter to allocate specific head dimensions to time, height, and width axes.
• Global indexing relies on absolute frame positions (f_start, f_end) rather than relative offsets, preventing temporal drift across sliding windows.
• Pre-cache application encodes temporal rotations into keys before they enter the KV cache, with the attention layer skipping re-application when enable_3d_rope is active.
• Architecture coordination occurs through a flag propagation chain from GCTBase through AggregatorStream to CameraCausalHead.

Frequently Asked Questions

What is the difference between 2D and 3D RoPE in LingBot-Map?

2D RoPE treats height and width as independent spatial axes by concatenating separate rotary embeddings for each. 3D RoPE introduces a temporal component that reserves specific head dimensions for frame indices, enabling the model to distinguish temporal positions using absolute global time rather than just spatial coordinates.

Why does temporal consistency matter for streaming SLAM?

Streaming SLAM processes video incrementally using a KV cache to avoid recomputing past states. Without temporal consistency, the same physical frame processed in different sliding windows would receive different positional encodings, causing the model to treat identical scenes as distinct events and accumulating positional drift over time.

How does 3D RoPE interact with the KV cache?

3D RoPE applies temporal rotations to keys before they are stored in the KV cache, as implemented in lingbot_map/layers/attention.py. When retrieving cached keys, the attention mechanism checks the enable_3d_rope flag and skips re-applying rotary embeddings. This ensures all cached keys remain on the same global temporal coordinate system regardless of when they were computed.

Where is the temporal dimension size configured?

The allocation between temporal, height, and width dimensions is configured via the fhw_dim parameter in WanRotaryPosEmbed.__init__ (default [20, 22, 22] in lingbot_map/layers/rope.py). These values must sum to the total attention_head_dim, with the first element representing the temporal allocation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}