# How 3D RoPE Enables Temporal Consistency in Streaming Inference for Real-Time SLAM

> 3D RoPE ensures temporal consistency in streaming inference for real-time SLAM by encoding global temporal indices with spatial coordinates, preventing positional drift.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: deep-dive
- Published: 2026-07-04

---

**LingBot-Map leverages a 3-D Rotary Position Embedding (3-D RoPE) that encodes global temporal indices alongside spatial coordinates, ensuring that attention scores remain stable across arbitrary streaming batches and preventing positional drift during incremental frame processing.**

LingBot-Map is an open-source SLAM framework designed for streaming video understanding. Unlike standard vision transformers that process fixed batches, streaming inference requires handling frames that arrive sequentially with variable timing. The implementation solves the temporal consistency problem by extending traditional 2-D RoPE with a dedicated **time dimension** and **global frame indexing**, as implemented in the `WanRotaryPosEmbed` class and integrated through the `AggregatorStream` module.

## Separate Frequency Tables for Time, Height, and Width

The foundation of temporal consistency lies in decomposing position encoding across three independent axes. In [`lingbot_map/layers/rope.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/rope.py), the `WanRotaryPosEmbed` class initializes separate 1-D frequency tensors for **time (t)**, **height (h)**, and **width (w)** using the `get_1d_rotary_pos_embed` helper.

During initialization (lines 94-112), the system concatenates these three frequency matrices into a unified representation. This design ensures that a token’s rotational encoding is a deterministic function of its absolute coordinates in the video volume, not merely its local batch position. The concatenation allows the model to attend to spatiotemporal relationships without ambiguity between spatial and temporal dimensions.

## Global Frame Indexing for Absolute Consistency

The critical innovation that prevents drift is the use of **global frame indices** rather than local batch offsets. The `AggregatorStream` class in [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py) tracks the absolute frame counters `f_start` and `f_end` during streaming.

When generating positions via `_get_3d_positions_streaming` (lines 66-76), the method passes these global indices to the RoPE generator:

```python
pos3d = self.rope3d(
    ppf=num_frames,
    pph=H // self.patch_size,
    ppw=W // self.patch_size,
    patch_start_idx=self.num_special_tokens,
    device=device,
    f_start=f_start,  # Global start frame

    f_end=f_end,      # Exclusive global end

)

```

This guarantees that frame *t = 5* receives an identical positional encoding regardless of when it enters the processing pipeline. Without global indexing, the same physical frame processed in different batches would receive incompatible encodings, breaking temporal coherence.

## Real-Arithmetic Embedding into the KV Cache

Once computed, the 3-D RoPE tensor (shape `[1, 1, N, head_dim//2]`) must be integrated into the attention mechanism without disrupting the **KV cache** consistency. LingBot-Map applies the rotary embedding using a **real-arithmetic implementation** (avoiding complex view operations) via `apply_rotary_emb` in [`lingbot_map/layers/rope.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/rope.py) (lines 35-48).

The embedding is baked into key/value tensors **before** storage:

```python
kv = apply_rotary_emb(k, pos3d)  # Real-arithmetic rotation

```

Because the KV cache stores already-rotated keys, subsequent frames attend to temporally consistent representations. When frame *t = 10* queries past frames, the cached keys for frame *t = 5* retain their original absolute positional encoding, enabling accurate temporal attention scores regardless of intervening processing steps.

## Causal Attention and Sliding-Window Compatibility

The 3-D RoPE design naturally supports **causal attention** and **sliding-window** mechanisms. Since each token embeds its absolute temporal index via `enable_3d_rope` (configured in `AggregatorStream.__init__`, lines 34-44), the causal mask operates on a stable coordinate system. The model can drop older KV entries to respect memory constraints without re-indexing positions, as the relative temporal relationships (`t_i < t_j`) remain implicit in the rotary encodings.

## Implementation Example: Enabling Streaming 3D RoPE

To activate temporal consistency in your streaming pipeline, initialize the model with the 3-D RoPE flag enabled:

```python

# example_01_streaming_setup.py

import torch
from lingbot_map.models.gct_stream import GCTStream

# Build model with temporal consistency

model = GCTStream(
    embed_dim=256,
    depth=12,
    num_heads=16,
    patch_size=16,
    enable_3d_rope=True,  # Activate 3-D temporal RoPE

    max_frame_num=1024,   # Maximum supported frames

)

```

During inference, the `AggregatorStream` automatically handles global indexing. You can verify the position generation by inspecting the internal state:

```python

# example_03_inspecting_rope_positions.py

from lingbot_map.aggregator.stream import AggregatorStream

agg = AggregatorStream(
    enable_3d_rope=True,
    max_frame_num=1024,
    patch_size=16,
    embed_dim=256
)

# Simulate frames 5-7 (global indices)

pos3d = agg._get_3d_positions_streaming(
    num_frames=3,
    H=480, W=640,
    device=torch.device('cpu'),
    f_start=5,
    f_end=8
)
print(f"3-D RoPE shape: {pos3d.shape}")  # (1, 1, N, head_dim//2)

```

## Key Source Files

| File | Component | Purpose |
|------|-----------|---------|
| [`lingbot_map/layers/rope.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/rope.py) | `WanRotaryPosEmbed`, `apply_rotary_emb` | Generates 3-D frequencies and applies real-arithmetic rotation |
| [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py) | `_init_3d_rope`, `_get_3d_positions_streaming` | Manages global frame indices and RoPE lifecycle |
| [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py) | `GCTStream` | Exposes `enable_3d_rope` configuration to top-level API |

## Summary

- **3-D RoPE** extends 2-D spatial encoding with a time dimension using concatenated frequency tables for temporal, height, and width axes.
- **Global frame indices** (`f_start`, `f_end`) ensure tokens maintain consistent encodings across arbitrary streaming batches, preventing drift.
- **Real-arithmetic application** bakes positions into the KV cache before storage, ensuring later frames attend to stable historical representations.
- The implementation supports **causal attention** and **sliding-window** contexts without requiring positional re-indexing.

## Frequently Asked Questions

### What distinguishes 3D RoPE from standard 2D RoPE implementations?

Standard 2D RoPE encodes only spatial height and width dimensions, making it suitable for static images but insufficient for video sequences. LingBot-Map's 3D RoPE adds a **time dimension** by concatenating separate frequency tables for temporal, height, and width axes, allowing the attention mechanism to distinguish between tokens from different frames while maintaining geometric relationships.

### Why must streaming inference use global frame indices rather than local batch positions?

Using local batch indices would assign position 0 to the first frame of every batch, causing frame *t = 100* to receive a different encoding when processed in isolation versus within a full sequence. By passing **global indices** (`f_start`, `f_end`) to `WanRotaryPosEmbed`, LingBot-Map ensures each physical frame receives a deterministic encoding based on its absolute temporal location in the video stream.

### How does 3D RoPE interact with sliding-window attention constraints?

Since 3D RoPE embeds **absolute temporal coordinates** into each token, the causal mask can correctly identify temporal relationships even when the KV cache evicts old frames. When sliding-window attention drops historical frames, the remaining entries retain their original absolute positions, so relative temporal ordering (`t_i < t_j`) remains valid without re-computation or re-indexing.

### Why does LingBot-Map use real-arithmetic instead of complex tensors for RoPE application?

The implementation in `apply_rotary_emb` uses **real-arithmetic** to avoid complex view operations that can introduce numerical instability or incompatibility with certain inference engines. This approach mixes the pre-computed frequency components directly with key/value vectors using real-valued operations before storage in the FlashInfer or SDPA KV cache, ensuring efficient and numerically stable temporal consistency across streaming steps.