How LingBot-Map Implements 3D RoPE for Temporal Consistency

LingBot-Map implements 3D RoPE by extending 2D rotary position embeddings with a temporal dimension in WanRotaryPosEmbed, applying global frame indices to tokens before they enter the KV cache to ensure consistent positional encoding across streaming inference.

LingBot-Map is an open-source transformer-based architecture designed for streaming SLAM and map-building tasks. To maintain temporal consistency across video frames, the repository implements a specialized 3D Rotary Position Embedding (3D RoPE) mechanism that treats time, height, and width as independent rotational axes. This article examines how the WanRotaryPosEmbed class in lingbot_map/layers/rope.py leverages global frame indexing to prevent temporal drift during causal inference.

What Is 3D RoPE in LingBot-Map?

Extending 2D Rotary Position Embeddings

Traditional 2D RoPE encodes token positions by rotating feature vectors using sinusoidal matrices based on spatial coordinates. In lingbot_map/layers/rope.py, the 2D implementation treats height and width independently, concatenating their rotated components. 3D RoPE introduces a third rotational axis for the temporal dimension, assigning each video frame a distinct rotational signature based on its absolute position in the sequence.

The WanRotaryPosEmbed Class Architecture

The core implementation resides in the WanRotaryPosEmbed class. The constructor receives the attention head dimension and splits it into three sub-dimensions for time (t_dim), height (h_dim), and width (w_dim), ensuring that t_dim + h_dim + w_dim = head_dim.

# lingbot_map/layers/rope.py

class WanRotaryPosEmbed(nn.Module):
    """3D rotary position embedding (t, h, w)."""
    def __init__(self, attention_head_dim, patch_size, max_seq_len=1024,
                 theta=10000.0, fhw_dim=[20, 22, 22]):
        # Splits head_dim into t-, h-, w-sub-dimensions

        # Pre-computes separate 1-D RoPE tables for each axis

        # Stores complex-valued frequencies in self.freqs

The class pre-computes frequency tables for each axis using get_1d_rotary_pos_embed, concatenating them into self.freqs for efficient indexing during inference.

How 3D RoPE Enables Temporal Consistency

Global Frame Indexing vs. Relative Positioning

Temporal consistency requires that the same global frame index generates identical positional embeddings regardless of sliding window position. Unlike relative window indexing—which causes drift as frames shift through the buffer—3D RoPE uses absolute frame indices (f_start, f_end) to generate deterministic rotations. When the CameraCausalHead processes a chunk of frames, it calls WanRotaryPosEmbed with the global frame position:

# lingbot_map/heads/camera_head.py (excerpt)

if self.rope3d is not None:
    pos3d = self.rope3d(
        ppf=S,               # Number of frames in current chunk

        pph=1, ppw=1,        # Camera token occupies 1×1 patch

        patch_start_idx=0,
        device=pose_tokens.device,
        f_start=self.frame_idx,         # Global start frame

        f_end=self.frame_idx + S,       # Global end frame

    )

This ensures that frame 100 always receives the same rotational encoding whether processed in isolation or as part of a sliding window.

The KV Cache Integration

The positional embedding applies before keys enter the KV cache. In lingbot_map/layers/attention.py, the attention block checks if enable_3d_rope and skips standard 2D RoPE application when the 3D variant is active. This prevents double-application of rotations when retrieving cached keys.

Because the temporal rotation is embedded at the moment of key creation, cached representations maintain their global temporal alignment. When the model queries past frames, the stored keys already contain the correct temporal rotation, ensuring consistent attention scores across streaming steps.

Flag Propagation and Component Initialization

The enable_3d_rope flag propagates through the model hierarchy:

1. GCTBase (lingbot_map/models/gct_base.py, line 56) receives the configuration
2. GCTStream and AggregatorStream inherit the flag
3. AggregatorStream initializes the 3D RoPE module via _init_3d_rope() (lines 101-104 of lingbot_map/aggregator/stream.py)
4. CameraCausalHead instantiates the embedding layer during initialization (lines 86-108 of lingbot_map/heads/camera_head.py)

This cascading configuration ensures that all components agree on whether to use temporal position encoding.

Implementation Flow: From Model Config to Attention

The temporal consistency mechanism follows a strict initialization and execution pipeline:

1. Model Configuration – Set enable_3d_rope=True when instantiating GCTStream or GCTBase
2. Aggregator Initialization – AggregatorStream creates the shared WanRotaryPosEmbed instance in _init_3d_rope()
3. Head Setup – CameraCausalHead receives the 3D RoPE module and stores it as self.rope3d
4. Position Generation – During each forward pass, the camera head generates position tensors using global frame indices
5. Attention Application – The attention block receives the pre-computed 3D positions and applies them to queries and keys before caching

Code Example: Enabling 3D RoPE in Practice

To utilize 3D RoPE for temporal consistency in your own inference pipeline:

import torch
from lingbot_map.models.gct_stream_window_v2 import GCTStream
from lingbot_map.utils.geometry import setup_dummy_input

# Initialize model with 3D RoPE enabled

model = GCTStream(
    img_size=518,
    patch_size=14,
    embed_dim=1024,
    disable_global_rope=False,
    enable_3d_rope=True,  # Enable temporal consistency

    max_frame_num=1024,
)

# Create dummy streaming input (B=1, 3 frames, 1 camera token per frame)

tokens, mask = setup_dummy_input(num_frames=3)

# Forward pass with causal inference

with torch.no_grad():
    out = model(tokens, mask=mask, causal_inference=True)

# Output maintains temporal alignment across frames

print(out.shape)  # (B, S, output_dim)

The setup_dummy_input helper generates tensors of shape [B, S, C] where S represents the frame count, simulating the token embeddings produced by the vision backbone.

Key Source Files and Architecture

Component	File Path	Relevant Section
3D RoPE Implementation	lingbot_map/layers/rope.py	WanRotaryPosEmbed class definition (lines 79-106)
Camera Head Integration	lingbot_map/heads/camera_head.py	Initialization and forward pass position generation (lines 86-108)
Aggregator Stream	lingbot_map/aggregator/stream.py	_init_3d_rope() method (lines 98-104)
Base Model Configuration	lingbot_map/models/gct_base.py	enable_3d_rope constructor flag (lines 55-58)
Attention Mechanism	lingbot_map/layers/attention.py	Conditional RoPE application (lines 154-162)
CLI Demo	demo.py	Argument parsing for feature flag (lines 365-371)

Summary

• 3D RoPE extends 2D rotary embeddings by adding a temporal dimension through WanRotaryPosEmbed in lingbot_map/layers/rope.py
• Global frame indexing ensures identical positional encoding for the same frame regardless of sliding window position
• Pre-cache application applies temporal rotations before keys enter the KV cache, preventing alignment drift
• Flag propagation cascades from GCTBase through AggregatorStream to CameraCausalHead for coordinated activation
• Conditional attention blocks avoid double-application by checking enable_3d_rope before applying standard 2D RoPE

Frequently Asked Questions

What is the difference between 2D RoPE and 3D RoPE in LingBot-Map?

2D RoPE encodes spatial height and width dimensions by rotating feature vectors with sinusoidal matrices based on patch grid coordinates. 3D RoPE adds a temporal dimension that assigns frame-specific rotations based on absolute frame indices, enabling the model to maintain consistent temporal relationships across streaming video sequences.

Why must 3D RoPE be applied before the KV cache?

Applying 3D RoPE before caching ensures that stored keys contain the correct global temporal rotation. When the model retrieves cached keys during streaming inference, the temporal positions are already aligned with the query's current frame index. This prevents the "drift" that would occur if relative window positions were used or if rotations were applied twice.

How does LingBot-Map prevent double-application of rotary embeddings?

The attention blocks in lingbot_map/layers/attention.py check the enable_3d_rope flag before applying standard 2D RoPE. When 3D RoPE is enabled, the temporal and spatial rotations are handled externally by WanRotaryPosEmbed and passed to the attention mechanism, ensuring the rotary transformation is applied exactly once per token.

Can 3D RoPE be disabled for non-streaming inference?

Yes. The enable_3d_rope parameter in GCTBase and GCTStream controls this feature. When set to False, the model falls back to standard 2D RoPE, which is suitable for single-frame processing or scenarios where temporal consistency across cached frames is not required.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}