What Is Anchor Context in the Geometric Context Transformer and How Does It Help with Coordinate Grounding?

Anchor context is a fixed reference frame containing depth values that serves as a global geometric anchor, enabling the Geometric Context Transformer to ground coordinates by computing a robust median depth ratio that stabilizes metric-scale predictions and mitigates long-range drift.

The Geometric Context Transformer (GCT) is a streaming architecture in the Robbyant/lingbot-map repository that jointly reasons about camera pose, dense geometry, and long-range drift. A central mechanism enabling this multi-frame reasoning is anchor context, which provides a shared reference system that ties all streaming predictions together across video sequences.

What Is Anchor Context in GCT?

Anchor context consists of reference depth values sampled from a fixed frame within the video stream. These anchors serve as a global geometric reference that the model compares against all incoming frames, eliminating the need to re-establish a coordinate system from scratch for every new prediction.

The Anchor Depth Tensor

In the implementation found in lingbot_map/models/gct_stream_window_v2.py, the model receives an anchor_depth tensor as part of its forward pass inputs. This tensor contains the depth map from the anchored frame and is used to compute geometric relationships with current frame predictions.

The code tracks the anchor's position within the sliding window using an anchor_offset variable, ensuring the model knows which tokens correspond to the fixed reference frame.

Scale Estimation via Depth Ratios

The _depth_ratio_scale helper method computes a per-batch scaling factor by comparing the anchor depth against the current target depth. The implementation calculates the median ratio to ensure robustness against outliers:

# Simplified excerpt from GCTStream (≈ lines 800-810)

anchor_depth: torch.Tensor,
...

# Estimate per-batch scale as the median depth ratio anchor/target.

a = anchor_depth.to(torch.float32).reshape(batch_size, -1)
s_ab = torch.median(a / target_depth, dim=1).values

This median-based approach provides a reliable metric-scale factor that grounds the current frame's predictions to the anchor's coordinate system.

How Anchor Context Enables Coordinate Grounding

Anchor context improves coordinate grounding through three primary mechanisms that work together to maintain geometric consistency across long video sequences.

Stable Reference Frame

Because the anchor remains fixed throughout the streaming process, predictions from later frames can be directly compared to the same geometric reference. This eliminates scale ambiguity that would otherwise accumulate when processing frames independently.

Robust Scale Estimation

Using the median of depth ratios rather than the mean reduces sensitivity to outliers and noisy depth measurements. As implemented in the lingbot_map codebase, this robust statistical measure ensures that temporary occlusions or textureless regions do not destabilize the global scale estimate.

Drift Mitigation

The anchor information is stored in the trajectory memory, providing a long-range reference that corrects accumulated pose drift. This mechanism functions similarly to loop-closure in SLAM systems, repeatedly re-anchoring predictions to the original geometry to prevent error accumulation over time.

Implementation in the lingbot-map Repository

The anchor context mechanism is implemented across several key files in the Robbyant/lingbot-map repository. The base logic resides in lingbot_map/models/gct_base.py, while the streaming implementation with KV-cache handling appears in lingbot_map/models/gct_stream_window_v2.py.

Basic Usage Example

When running inference with the GCTStream class, you must provide the anchor_depth parameter to establish the reference frame:

import torch
from lingbot_map.models.gct_stream import GCTStream

# Load a pretrained streaming GCT model

model = GCTStream(...)
model.eval()

# Dummy inputs (replace with real image tensors and intrinsics)

images = torch.randn(4, 3, 480, 640)       # B×C×H×W

intrinsics = torch.randn(4, 4)             # B×4×4

pose_ref = torch.randn(4, 7)               # B×(quat+trans)

anchor_depth = torch.randn(4, 1, 480, 640) # Anchored depth map

# Forward pass with anchor_depth

with torch.no_grad():
    output = model(
        images=images,
        intrinsics=intrinsics,
        pose_ref=pose_ref,
        anchor_depth=anchor_depth,
    )

Inspecting the Computed Scale

You can directly compute the scale factor using the internal method that compares anchor and target depths:

# Inside the model (illustrative snippet)

scale = GCTStream._depth_ratio_scale(
    anchor_depth=anchor_depth,
    target_depth=output["depth"],   # depth predicted for the current frame

)
print("Estimated batch scale:", scale)

The returned scale is a 1-D tensor containing one value per batch element, aligning the current frame's metric scale to the anchored frame.

Visualizing Alignment

After applying the computed scale, you can verify that the current frame aligns with the anchor coordinate system:

import matplotlib.pyplot as plt

# Visualise anchored depth vs. current depth (after scaling)

scaled_current = output["depth"] * scale[:, None, None, None]

plt.figure(figsize=(8, 4))
plt.subplot(1, 2, 1)
plt.title("Anchor depth")
plt.imshow(anchor_depth[0, 0].cpu(), cmap="viridis")
plt.subplot(1, 2, 2)
plt.title("Scaled current depth")
plt.imshow(scaled_current[0, 0].cpu(), cmap="viridis")
plt.show()

The two depth maps should now be pixel-wise aligned, confirming successful coordinate grounding to the anchor context.

Summary

• Anchor context provides a fixed reference frame of depth values that serves as a global geometric backbone for the Geometric Context Transformer.
• The _depth_ratio_scale method in lingbot_map/models/gct_stream_window_v2.py computes a robust scale factor using the median ratio between anchor and target depths.
• By grounding predictions to a stable anchor, the model eliminates per-frame scale ambiguity and mitigates long-range drift through repeated re-anchoring to the original geometry.
• The anchor_offset tracking ensures the model maintains awareness of which tokens correspond to the reference frame within the sliding window architecture.

Frequently Asked Questions

What is the difference between anchor context and regular temporal context?

Regular temporal context typically involves the previous few frames in a sliding window, providing short-term continuity. Anchor context specifically refers to a fixed, globally consistent reference frame that persists throughout the entire sequence, serving as a geometric "origin" that all other frames are aligned to, rather than just relative to their immediate predecessors.

How does the median depth ratio calculation handle outliers?

The implementation uses torch.median(a / target_depth, dim=1).values rather than the mean, which naturally rejects outliers. Extreme values from occlusion boundaries or sensor noise have minimal impact on the median, ensuring the scale estimate remains stable even when portions of the depth map contain errors.

Where is anchor context stored in the GCT architecture?

The anchor context is stored in the trajectory memory alongside pose information. According to the lingbot_map implementation, the anchor depth tensor and its associated metadata (including the anchor_offset position marker) are maintained in the model's memory buffers, allowing the transformer to reference the fixed frame throughout the streaming process.

Can anchor context work with varying camera intrinsics?

The current implementation assumes that the anchor depth and incoming frames share the same camera intrinsics or that the depth maps have been pre-aligned. The scale computation relies on pixel-wise depth ratios, so consistent calibration between the anchor frame and target frames is required for accurate coordinate grounding.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}