How 3D RoPE Enables Temporal Consistency in Streaming Inference for Real-Time SLAM

LingBot-Map leverages a 3-D Rotary Position Embedding (3-D RoPE) that encodes global temporal indices alongside spatial coordinates, ensuring that attention scores remain stable across arbitrary streaming batches and preventing positional drift during incremental frame processing.

LingBot-Map is an open-source SLAM framework designed for streaming video understanding. Unlike standard vision transformers that process fixed batches, streaming inference requires handling frames that arrive sequentially with variable timing. The implementation solves the temporal consistency problem by extending traditional 2-D RoPE with a dedicated time dimension and global frame indexing, as implemented in the WanRotaryPosEmbed class and integrated through the AggregatorStream module.

Separate Frequency Tables for Time, Height, and Width

The foundation of temporal consistency lies in decomposing position encoding across three independent axes. In lingbot_map/layers/rope.py, the WanRotaryPosEmbed class initializes separate 1-D frequency tensors for time (t), height (h), and width (w) using the get_1d_rotary_pos_embed helper.

During initialization (lines 94-112), the system concatenates these three frequency matrices into a unified representation. This design ensures that a token’s rotational encoding is a deterministic function of its absolute coordinates in the video volume, not merely its local batch position. The concatenation allows the model to attend to spatiotemporal relationships without ambiguity between spatial and temporal dimensions.

Global Frame Indexing for Absolute Consistency

The critical innovation that prevents drift is the use of global frame indices rather than local batch offsets. The AggregatorStream class in lingbot_map/aggregator/stream.py tracks the absolute frame counters f_start and f_end during streaming.

When generating positions via _get_3d_positions_streaming (lines 66-76), the method passes these global indices to the RoPE generator:

pos3d = self.rope3d(
    ppf=num_frames,
    pph=H // self.patch_size,
    ppw=W // self.patch_size,
    patch_start_idx=self.num_special_tokens,
    device=device,
    f_start=f_start,  # Global start frame

    f_end=f_end,      # Exclusive global end

)

This guarantees that frame t = 5 receives an identical positional encoding regardless of when it enters the processing pipeline. Without global indexing, the same physical frame processed in different batches would receive incompatible encodings, breaking temporal coherence.

Real-Arithmetic Embedding into the KV Cache

Once computed, the 3-D RoPE tensor (shape [1, 1, N, head_dim//2]) must be integrated into the attention mechanism without disrupting the KV cache consistency. LingBot-Map applies the rotary embedding using a real-arithmetic implementation (avoiding complex view operations) via apply_rotary_emb in lingbot_map/layers/rope.py (lines 35-48).

The embedding is baked into key/value tensors before storage:

kv = apply_rotary_emb(k, pos3d)  # Real-arithmetic rotation

Because the KV cache stores already-rotated keys, subsequent frames attend to temporally consistent representations. When frame t = 10 queries past frames, the cached keys for frame t = 5 retain their original absolute positional encoding, enabling accurate temporal attention scores regardless of intervening processing steps.

Causal Attention and Sliding-Window Compatibility

The 3-D RoPE design naturally supports causal attention and sliding-window mechanisms. Since each token embeds its absolute temporal index via enable_3d_rope (configured in AggregatorStream.__init__, lines 34-44), the causal mask operates on a stable coordinate system. The model can drop older KV entries to respect memory constraints without re-indexing positions, as the relative temporal relationships (t_i < t_j) remain implicit in the rotary encodings.

Implementation Example: Enabling Streaming 3D RoPE

To activate temporal consistency in your streaming pipeline, initialize the model with the 3-D RoPE flag enabled:

# example_01_streaming_setup.py

import torch
from lingbot_map.models.gct_stream import GCTStream

# Build model with temporal consistency

model = GCTStream(
    embed_dim=256,
    depth=12,
    num_heads=16,
    patch_size=16,
    enable_3d_rope=True,  # Activate 3-D temporal RoPE

    max_frame_num=1024,   # Maximum supported frames

)

During inference, the AggregatorStream automatically handles global indexing. You can verify the position generation by inspecting the internal state:

# example_03_inspecting_rope_positions.py

from lingbot_map.aggregator.stream import AggregatorStream

agg = AggregatorStream(
    enable_3d_rope=True,
    max_frame_num=1024,
    patch_size=16,
    embed_dim=256
)

# Simulate frames 5-7 (global indices)

pos3d = agg._get_3d_positions_streaming(
    num_frames=3,
    H=480, W=640,
    device=torch.device('cpu'),
    f_start=5,
    f_end=8
)
print(f"3-D RoPE shape: {pos3d.shape}")  # (1, 1, N, head_dim//2)

Key Source Files

File	Component	Purpose
lingbot_map/layers/rope.py	WanRotaryPosEmbed, apply_rotary_emb	Generates 3-D frequencies and applies real-arithmetic rotation
lingbot_map/aggregator/stream.py	_init_3d_rope, _get_3d_positions_streaming	Manages global frame indices and RoPE lifecycle
lingbot_map/models/gct_stream.py	GCTStream	Exposes enable_3d_rope configuration to top-level API

Summary

• 3-D RoPE extends 2-D spatial encoding with a time dimension using concatenated frequency tables for temporal, height, and width axes.
• Global frame indices (f_start, f_end) ensure tokens maintain consistent encodings across arbitrary streaming batches, preventing drift.
• Real-arithmetic application bakes positions into the KV cache before storage, ensuring later frames attend to stable historical representations.
• The implementation supports causal attention and sliding-window contexts without requiring positional re-indexing.

Frequently Asked Questions

What distinguishes 3D RoPE from standard 2D RoPE implementations?

Standard 2D RoPE encodes only spatial height and width dimensions, making it suitable for static images but insufficient for video sequences. LingBot-Map's 3D RoPE adds a time dimension by concatenating separate frequency tables for temporal, height, and width axes, allowing the attention mechanism to distinguish between tokens from different frames while maintaining geometric relationships.

Why must streaming inference use global frame indices rather than local batch positions?

Using local batch indices would assign position 0 to the first frame of every batch, causing frame t = 100 to receive a different encoding when processed in isolation versus within a full sequence. By passing global indices (f_start, f_end) to WanRotaryPosEmbed, LingBot-Map ensures each physical frame receives a deterministic encoding based on its absolute temporal location in the video stream.

How does 3D RoPE interact with sliding-window attention constraints?

Since 3D RoPE embeds absolute temporal coordinates into each token, the causal mask can correctly identify temporal relationships even when the KV cache evicts old frames. When sliding-window attention drops historical frames, the remaining entries retain their original absolute positions, so relative temporal ordering (t_i < t_j) remains valid without re-computation or re-indexing.

Why does LingBot-Map use real-arithmetic instead of complex tensors for RoPE application?

The implementation in apply_rotary_emb uses real-arithmetic to avoid complex view operations that can introduce numerical instability or incompatibility with certain inference engines. This approach mixes the pre-computed frequency components directly with key/value vectors using real-valued operations before storage in the FlashInfer or SDPA KV cache, ensuring efficient and numerically stable temporal consistency across streaming steps.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}