How LingBot-Map Uses Anchor Tokens for Coordinate Grounding in Streaming Video

LingBot-Map aligns predictions across overlapping video windows by using shared key-frames as anchor tokens, applying a similarity transform derived from depth ratios to maintain global coordinate consistency.

The Robbyant/lingbot-map repository implements a streaming architecture for long video sequences that processes frames in manageable chunks while maintaining globally consistent 3-D reconstructions. To prevent drift when stitching consecutive windows, the system relies on anchor tokens—specific key-frames present in overlapping regions—to establish reference points that ground coordinates across the entire sequence.

Why Streaming Video Requires Coordinate Grounding

Processing long video streams in a single forward pass is often infeasible due to memory constraints. LingBot-Map solves this by splitting videos into overlapping windows, but each window initially produces predictions in its own local coordinate system. Without a common reference, accumulated errors would destroy the global reconstruction. Anchor tokens serve as fixed points that bridge adjacent windows, allowing the model to recover absolute scale and orientation through a similarity transformation.

The Three-Step Anchor Token Mechanism

The anchoring logic resides in lingbot_map/models/gct_stream_window_v2.py and operates when merging a previous window with the current one. The process leverages the is_keyframe flag to identify stable reference frames within the overlapping region.

Step 1: Detecting Paired Key-Frames in Overlaps

The model first identifies paired key-frames—frames marked as key-frames in both the tail of the previous window and the head of the current window. The latest such pair becomes the anchor.

# lingbot_map/models/gct_stream_window_v2.py#L528-L560

if is_kf_prev is not None and is_kf_curr is not None and eff_overlap > 0:
    kf_prev_tail = is_kf_prev[0, start:start + eff_overlap].to(torch.bool)
    kf_curr_head = is_kf_curr[0, :eff_overlap].to(torch.bool)
    paired_mask = kf_prev_tail & kf_curr_head
    if paired_mask.any():
        paired_offsets = torch.nonzero(paired_mask, as_tuple=False).flatten().to(device)

When pairs exist, the system selects the last offset in the overlap region as the anchor:

anchor_offset = int(paired_offsets[-1].item())
idx_a = start + anchor_offset      # index in previous window

idx_b = anchor_offset               # index in current window

Step 2: Extracting Pose and Depth at the Anchor

Using the indices idx_a and idx_b, the model retrieves pose encodings (pe_prev, pe_curr) and depth maps (da, db) for the anchor frames. The rotation and translation mapping the current window to the previous are computed from the poses, while the scale derives from the depth ratio between the two anchor views:

# lingbot_map/models/gct_stream_window_v2.py#L802-L815

s_ab = GCTStream._depth_ratio_scale(
    da[:, prev_depth_idx, ..., 0],
    db[:, curr_depth_idx, ..., 0],
    batch_size, device,
)

This _depth_ratio_scale method calculates the relative scale factor necessary to align the metric spaces of the two windows.

Step 3: Applying the Similarity Transform

Finally, the similarity transform (s, R, t)—comprising scale, rotation, and translation—is applied to every prediction tensor in the current window. This warps pose encodings, depth values, and world points into the coordinate system of the previous window:

# lingbot_map/models/gct_stream_window_v2.py#L840-L878

warped["pose_enc"] = ...  # R and t applied to rotation & centre

warped["depth"] = d * s.view(batch_size, 1, 1, 1, 1)
warped["world_points"] = ...  # s·R·p + t

By repeatedly applying this procedure across all windows, LingBot-Map constructs a globally coherent 3-D reconstruction while bounding the KV-cache size.

Implementation Example

Users interact with this mechanism through the GCTStream class, which handles anchor selection internally during windowed inference. The following example demonstrates how window and overlap parameters influence anchor behavior:

import torch
from lingbot_map.models.gct_stream_window_v2 import GCTStream

# Initialize model with streaming support

model = GCTStream(
    img_size=518,
    patch_size=14,
    embed_dim=1024,
    enable_3d_rope=True,          # Required for temporal consistency

    kv_cache_sliding_window=64,    # Maintains recent KV entries

)

# Dummy video: Batch=1, 120 frames, 3 channels, 518x518 resolution

video = torch.rand(1, 120, 3, 518, 518)

# Windowed inference with automatic anchor alignment

pred = model.inference_windowed(
    images=video,
    window_size=16,                # 16 key-frames per window

    overlap_size=4,                # 4 frames shared between windows

    keyframe_interval=2,           # Every 2nd frame enters KV cache

)

# Results are globally aligned through anchor tokens

print(pred["pose_enc"].shape)      # torch.Size([1, 120, 9])

print(pred["world_points"].shape)  # torch.Size([1, 120, 518, 518, 3])

The window_size and overlap_size parameters directly control how many frames are shared between consecutive windows, determining the available pool for anchor token selection.

Key Source Files

• lingbot_map/models/gct_stream_window_v2.py – Contains the _pairwise_alignment logic, anchor selection (lines 528-560), _depth_ratio_scale computation (lines 802-815), and similarity transform application (lines 840-878).
• lingbot_map/utils/pose_enc.py – Utilities for converting pose encodings into extrinsic and intrinsic matrices used during anchor pose extraction.
• lingbot_map/aggregator/stream.py – Implements the KV-cache-enabled aggregator that maintains sliding-window tokens, including anchor tokens across window boundaries.

Summary

• Anchor tokens are shared key-frames in overlapping window regions that serve as reference points for coordinate alignment.
• The system detects paired key-frames using boolean masks on the is_keyframe tensor, selecting the last valid pair as the anchor.
• Scale is computed via _depth_ratio_scale comparing depth values at the anchor positions, while rotation and translation derive from pose encodings.
• A similarity transform (s, R, t) warps all predictions in the current window to match the previous window's coordinate system.
• This mechanism enables processing of arbitrarily long videos while maintaining global metric consistency and bounded memory usage.

Frequently Asked Questions

What are anchor tokens in LingBot-Map?

Anchor tokens are specific key-frames identified in the overlapping regions between consecutive processing windows. They act as fixed reference points that exist in both the tail of the previous window and the head of the current window, allowing the model to establish a correspondence between the two local coordinate systems.

How does the depth ratio calculate scale between windows?

The _depth_ratio_scale method computes the relative scale factor by comparing depth values at the anchor frame indices. It takes the depth map from the previous window's anchor (da) and the current window's anchor (db), then calculates the ratio that aligns these measurements, effectively recovering the metric scale needed to stitch the windows together.

Why use overlapping windows instead of processing the entire video?

Overlapping windows enable processing of arbitrarily long video streams while keeping GPU memory usage bounded. The KV-cache only retains recent tokens up to a fixed window size, and the overlap provides the necessary redundancy to find anchor tokens that bridge adjacent segments without requiring the entire sequence to reside in memory simultaneously.

What parameters control anchor token selection?

Anchor behavior is governed primarily by window_size (frames per chunk), overlap_size (shared frames between chunks), and keyframe_interval (frequency of key-frame sampling). Increasing overlap_size provides more potential anchor candidates but increases computational cost, while keyframe_interval determines how densely the anchor search samples the temporal dimension.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}