# What Is the Pose‑Reference Window in the Geometric Context Transformer?

> Discover how the pose-reference window in Geometric Context Transformers caches recent pose encodings to condition predictions on local trajectory context and correct drift during streaming 3D reconstruction.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: deep-dive
- Published: 2026-07-08

---

**The pose‑reference window functions as a short‑term sliding memory buffer that caches recent pose encodings in the KV cache, allowing the transformer to condition current predictions on local trajectory context and correct drift during streaming 3D reconstruction.**

The Geometric Context Transformer (GCT) in the `Robbyant/lingbot-map` repository achieves robust streaming 3D reconstruction by unifying three complementary mechanisms: anchor context, trajectory memory, and the pose‑reference window. This article explores how the pose‑reference window supplies the transformer with critical local pose information to stabilize estimates and enable efficient keyframe handling.

## How the Pose‑Reference Window Works

The pose‑reference window serves as the engine that supplies the transformer with local pose information during streaming inference. Unlike the long‑range trajectory memory that stores past camera poses for global alignment, the pose‑reference window maintains a short‑term sliding buffer of the most recent N frames—where N defaults to the `kv_cache_sliding_window` parameter (typically 64).

### Local Consistency Through Cached Pose Encodings

During each inference step, the transformer processes incoming frames alongside the KV cache of previous frames stored in the pose‑reference window. These cached pose encodings are treated as additional tokens that self‑attention layers can directly attend to, enabling the model to:

- **Condition current pose predictions on recent trajectory context**—the network compares the current view against the last few poses, reducing jitter and stabilizing estimates.
- **Facilitate drift correction**—by referencing nearby poses, the model detects and compensates for accumulated error before it propagates to subsequent frames.
- **Optimize memory usage**—non‑keyframes can query the pose‑reference window without storing their own KV entries, significantly reducing memory overhead.

## Implementation in the Lingbot‑Map Codebase

The pose‑reference window is implemented through the KV‑cache mechanisms of FlashInfer/SDPA, with specific logic controlling which frames enter the cache and how the window slides forward.

### Sliding Window Configuration

In [`lingbot_map/models/gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py) and its refined version [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py), the `kv_cache_sliding_window` parameter defines the window size (default 64). The window operates through several key mechanisms:

- **`_set_skip_append`**—prevents non‑keyframes from being added to the KV cache while still allowing them to attend to existing entries.
- **`_set_defer_eviction`**—controls when cached entries are removed from the window.
- **`keyframe_interval`**—determines how frequently frames are stored; non‑keyframes query the pose‑reference window without expanding it.

### Windowed Inference and Alignment

For long sequences, the `inference_windowed` method in `GCTStream` stitches together multiple windows, each anchored by a pose‑reference window. The `_pairwise_alignment` function aligns these windows with scale estimation to maintain global consistency across the entire sequence.

The underlying FlashInfer/SDPA KV cache is constructed in [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py), which manages the storage and retrieval of pose tokens that populate the reference window.

## Configuring the Pose‑Reference Window

Practical usage involves setting the sliding window size and keyframe interval to balance accuracy against memory constraints.

Run streaming inference with a 64‑frame pose‑reference window:

```python
from lingbot_map.models.gct_stream import GCTStream
import torch

model = GCTStream(
    kv_cache_sliding_window=64,      # size of the pose‑reference window

    keyframe_interval=2,            # store only every 2nd frame in the cache

    enable_3d_rope=True,            # keep temporal consistency

)

images = torch.randn(1, 1200, 3, 378, 518)  # [B, S, C, H, W]

preds = model.inference_streaming(
    images,
    keyframe_interval=2,   # non‑keyframes query the pose‑reference window but are not cached

    flow_threshold=0.0,    # use fixed‑interval keyframe mode

)

# Access the pose encodings for the recent window (last 64 frames)

recent_pose_enc = preds["pose_enc"][:, -64:]   # shape [B, 64, 9]

print("Recent pose‑reference window shape:", recent_pose_enc.shape)

```

Run windowed inference on long video sequences:

```bash
python demo.py \
  --model_path path/to/lingbot-map-long.pt \
  --video_path long_video.mp4 \
  --mode windowed \
  --window_size 128 \
  --overlap_keyframes 16 \
  --keyframe_interval 1   # each frame becomes a keyframe; the pose‑reference window holds 128 recent poses

```

## Summary

- The pose‑reference window acts as short‑term memory for the Geometric Context Transformer, caching recent pose encodings in the KV cache.
- It enables local consistency by allowing the model to condition predictions on the recent trajectory, reducing jitter and correcting drift.
- Non‑keyframes can query the window without being cached, enabling efficient memory usage through `keyframe_interval` configuration.
- Implementation resides in [`lingbot_map/models/gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window.py) and [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py), utilizing FlashInfer/SDPA for KV‑cache management.
- Key parameters include `kv_cache_sliding_window` (default 64), `_set_skip_append`, and `_set_defer_eviction`.

## Frequently Asked Questions

### What is the difference between the pose‑reference window and trajectory memory?

The pose‑reference window is a short‑term sliding buffer (typically 64 frames) that provides immediate local context for pose prediction, while trajectory memory is a long‑range cache that stores historical camera poses for global drift‑free alignment. The window handles local consistency, whereas trajectory memory handles global consistency.

### How does the pose‑reference window reduce memory usage during inference?

The window reduces memory by allowing non‑keyframes to attend to cached pose encodings without storing their own KV entries. By setting `keyframe_interval` greater than 1, only selected frames are appended to the cache via `_set_skip_append`, while intermediate frames query the existing window.

### What happens when the pose‑reference window reaches its maximum size?

When the cache reaches `kv_cache_sliding_window` entries, older frames are evicted according to the `_set_defer_eviction` logic. For long videos, the `inference_windowed` method processes overlapping windows and uses `_pairwise_alignment` to stitch them together, ensuring continuity across window boundaries.

### Can I adjust the size of the pose‑reference window?

Yes, you can configure the window size by setting the `kv_cache_sliding_window` parameter when initializing `GCTStream`. The default is 64 frames, but you can increase this for greater local context or decrease it to reduce memory consumption, depending on your hardware constraints and accuracy requirements.