# How Iterative Refinement Improves Pose Accuracy in LingBot-Map's Camera Head

> Learn how iterative refinement boosts LingBot-Map's camera head pose accuracy by progressively correcting estimates through a learned loop. Discover the trade-off between computation and precision.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: deep-dive
- Published: 2026-07-03

---

**Iterative refinement in LingBot-Map's Camera Head progressively corrects camera pose estimates through a learned loop that embeds previous predictions, modulates transformer tokens, and accumulates delta corrections over multiple steps, trading computation for precision.**

The **Camera Head** in the Robbyant/lingbot-map repository predicts camera extrinsics and intrinsics from transformer features using a coarse-to-fine optimization strategy. By default, the model runs **four iterative refinement steps** that progressively reduce translation and rotation errors, allowing users to balance between inference speed and pose accuracy by adjusting the `num_iterations` parameter.

## Understanding the Camera Head Architecture

The **CameraHead** and **CameraCausalHead** classes implement the iterative pose prediction logic in [`lingbot_map/heads/camera_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/camera_head.py). These heads process transformer token features to output camera translation, rotation (quaternion), and field-of-view parameters.

### The Refinement Loop in trunk_fn

The core refinement mechanism resides in the `trunk_fn` method (lines 99-112 for `CameraHead` and lines 96-112 for `CameraCausalHead`). This method implements a configurable loop controlled by `self.num_iterations`, which defaults to 4 but can be reduced to 1 for faster inference.

### Pose Embedding and Token Modulation

Each iteration begins by embedding the current pose estimate using `self.embed_pose`. The first iteration uses a learned "empty" pose token, while subsequent iterations embed the accumulated prediction from the previous step. This embedding generates modulation parameters (shift, scale, gate) via `self.poseLN_modulation`, which are applied to the camera token through `self.adaln_norm` and modulation operations.

## The Six-Step Refinement Process

The iterative refinement improves pose accuracy through six distinct operations repeated for `num_iterations`:

- **Embed the current pose estimate.** The loop initializes with a learned empty token, then uses the previous iteration's output.
- **Generate modulation parameters.** The `self.poseLN_modulation` layer produces shift, scale, and gate values from the pose embedding.
- **Adaptively normalize the camera token.** The `self.adaln_norm` layer combined with modulation allows the transformer to focus on residual errors.
- **Process through transformer blocks.** The modulated token passes through `self.trunk`, capturing richer contextual information from visual tokens.
- **Predict and accumulate delta poses.** A small MLP (`self.pose_branch`) predicts pose corrections, which are added to the current estimate (`pred_pose_enc = pred_pose_enc + pred_pose_enc_delta`).
- **Apply final activations.** The `activate_pose` function (defined in [`lingbot_map/heads/head_act.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/head_act.py)) converts raw predictions into valid translation, quaternion, and FOV values.

## Configuring Iteration Count for Speed vs. Accuracy

The `num_iterations` parameter provides direct control over the inference-accuracy trade-off. Setting `num_iterations=1` enables fast inference with coarse pose estimates, while the default of 4 iterations yields optimal pose quality on benchmarks such as TUM, Oxford, and KITTI.

## Code Implementation

The following example demonstrates how to instantiate the `CameraCausalHead` with iterative refinement and inspect the pose predictions:

```python
import torch
from lingbot_map.heads.camera_head import CameraCausalHead

# Create dummy token tensor (B, N, C)

dummy_tokens = torch.randn(2, 128, 2048)  # B=2, N=128 tokens, C=2048 features

tokens_list = [dummy_tokens]  # Model expects a list; last entry is used

# Instantiate with 4 refinement iterations (default)

camera_head = CameraCausalHead(
    dim_in=2048,
    trunk_depth=4,
    num_heads=16,
    num_iterations=4,  # Higher values improve accuracy at the cost of speed

)

# Forward pass returns list of pose predictions (one per iteration)

pose_predictions = camera_head(tokens_list)

# Each entry is tensor of shape (B, 9) = [Tx,Ty,Tz, Qx,Qy,Qz,Qw, FoVx, FoVy]

final_pose = pose_predictions[-1]
print(final_pose.shape)  # -> torch.Size([2, 9])

```

To reduce computation for faster inference, modify the iteration count:

```python
camera_head.num_iterations = 1  # Fast inference, reduced accuracy

pose_predictions = camera_head(tokens_list)

```

## Summary

- **Iterative refinement** in `CameraHead.trunk_fn` progressively corrects pose estimates through a learned loop, modeling non-linear pose manifolds through shallow refinements rather than single large jumps.
- The process embeds previous predictions, modulates transformer tokens via adaptive normalization, and accumulates delta corrections predicted by a lightweight MLP.
- Configuration of `num_iterations` allows explicit trade-offs between inference speed and pose accuracy, with 4 iterations providing optimal results on standard benchmarks.
- Key implementation files include [`lingbot_map/heads/camera_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/camera_head.py) for the refinement logic and [`lingbot_map/heads/head_act.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/head_act.py) for pose activation functions.

## Frequently Asked Questions

### How many iterations does LingBot-Map use by default?

The `CameraHead` and `CameraCausalHead` classes default to `num_iterations=4`, which provides the best pose accuracy on evaluation datasets like TUM, Oxford, and KITTI. Users can reduce this to 1 for faster inference at the cost of precision.

### What pose parameters does the Camera Head output?

The head outputs a 9-dimensional vector containing translation `[Tx, Ty, Tz]`, rotation quaternion `[Qx, Qy, Qz, Qw]`, and field-of-view `[FoVx, FoVy]`. The `activate_pose` function in [`lingbot_map/heads/head_act.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/head_act.py) ensures these values are properly constrained.

### Where is the iterative refinement loop implemented?

The refinement logic resides in the `trunk_fn` method of both `CameraHead` (lines 99-112) and `CameraCausalHead` (lines 96-112) within [`lingbot_map/heads/camera_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/camera_head.py). This method manages the embedding, modulation, and delta prediction steps across iterations.

### Can I use iterative refinement with streaming models?

Yes. The `CameraCausalHead` variant supports causal attention masks for streaming applications while maintaining the same iterative refinement structure. Integration examples are available in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py).