# How Trajectory AUC Calculation Works: Micro vs Macro Aggregation in Lingbot‑Map

> Understand trajectory AUC calculation in lingbot-map. Learn how micro vs macro aggregation pools rotation and translation errors to compute area under the curve.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: deep-dive
- Published: 2026-07-02

---

**Trajectory AUC calculation in lingbot‑map pools per‑scene rotation and translation errors to compute the area under the cumulative‑distribution curve, either globally via micro‑aggregation or as averaged per‑scene scores via macro‑aggregation.**

The `Robbyant/lingbot-map` repository provides a comprehensive benchmarking toolkit for SLAM and visual odometry systems. At its core lies a **trajectory AUC calculation** that quantifies trajectory accuracy by measuring relative pose errors across all frame pairs. This metric supports two distinct aggregation strategies—micro and macro—that handle multi‑scene datasets differently depending on whether you want to weight scenes by their size or treat them equally.

## The Three Stages of Trajectory AUC Calculation

The AUC pipeline implemented in [`benchmark/benchmark/evaluation/auc.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/evaluation/auc.py) processes trajectories through three distinct logical phases before reaching the final metric.

### Pose‑Pair Preparation

First, ground‑truth and predicted trajectories are loaded and aligned. The `AUCEvaluator.evaluate` method calls `_filter_valid_pose_pairs` (defined in [`benchmark/benchmark/evaluation/trajectory.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/evaluation/trajectory.py)) to ensure only valid corresponding poses remain. The system transforms coordinates to the world‑to‑camera (`W2C`) convention using utilities from [`benchmark/benchmark/geometry/transform.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/geometry/transform.py).

### Relative‑Pose Error Computation

For every unordered pair of valid frames, the system computes relative transformation errors. The helper `_build_pair_index` generates all frame combinations, while `_se3_to_relative_pose_error` calculates:
- **Rotation error** via `_rotation_angle`
- **Translation error** via `_translation_angle`

These functions appear around lines 57–68 in the source, processing SE(3) transforms to produce angular deviations in degrees.

### AUC and Accuracy Metrics

Using default angle thresholds of **3°, 5°, 15°, and 30°**, the evaluator computes **Racc** (rotation accuracy) and **Tacc** (translation accuracy) as the fraction of pairs with errors below each threshold. The core **trajectory AUC calculation** happens in `_calculate_auc_np` (lines 71–81), which integrates the cumulative distribution of the maximum error (rotation or translation) across all pairs.

## Micro vs Macro Aggregation

When benchmarking datasets containing multiple scenes, the repository offers two aggregation modes that answer different statistical questions.

### Micro Aggregation: Weight by Pair Count

**Micro aggregation** pools all raw rotation (`rError`) and translation (`tError`) values from every scene into single massive arrays. It then computes one global AUC, Racc, and Tacc across this entire pool.

- **Implementation**: `AUCEvaluator.aggregate_micro` (lines 14–25 in [`auc.py`](https://github.com/Robbyant/lingbot-map/blob/main/auc.py))
- **Behavior**: Scenes with more frame pairs contribute proportionally more to the final score
- **Use case**: Prioritizing overall system performance where larger sequences naturally carry more weight

The method concatenates error arrays using `np.concatenate` before calling `_calculate_auc_np`, ensuring the error distribution reflects the entire dataset’s raw data.

### Macro Aggregation: Equal Scene Weighting

**Macro aggregation** computes AUC, Racc, and Tacc independently for each scene, then averages these per‑scene scores using simple arithmetic mean.

- **Implementation**: `AUCEvaluator.aggregate_macro` (lines 27–35 in [`auc.py`](https://github.com/Robbyant/lingbot-map/blob/main/auc.py))
- **Behavior**: Every scene contributes equally regardless of frame count
- **Use case**: Fair comparison across benchmarks where short and long sequences should have equal influence

This approach prevents a single long trajectory from dominating the final metric, making it ideal for standardized leaderboards.

## Implementation Details

Both aggregation strategies reuse the same low‑level math. The `_calculate_auc_np` function accepts error arrays and thresholds, returning the area under the curve and threshold‑accuracies. The key difference lies in input preparation:

- **Micro**: Concatenates `rError` and `tError` lists from all scenes before calculation
- **Macro**: Calls `_calculate_auc_np` per scene, stores the results, then applies `np.mean` across scenes

Default thresholds are defined as `[3, 5, 15, 30]` degrees, though these can be customized during `AUCEvaluator` instantiation.

## Practical Code Examples

### Evaluate a Single Scene

```python
from benchmark.benchmark.io.trajectory import BSSLoader
from benchmark.benchmark.evaluation.auc import AUCEvaluator

gt_loader = BSSLoader("<gt_folder>")
pred_loader = BSSLoader("<pred_folder>")

evaluator = AUCEvaluator()  # Default thresholds: [3, 5, 15, 30]

scene_metrics = evaluator.evaluate(gt_loader, pred_loader)

print("Scene AUC @ 5°:", scene_metrics["AUC_05"])
print("Scene Racc @ 15°:", scene_metrics["Racc_15"])

```

The `evaluate` method returns a dictionary containing raw error arrays (`rError`, `tError`) that subsequent aggregation functions consume.

### Micro Aggregation Across Scenes

```python
from pathlib import Path
from benchmark.benchmark.evaluation.auc import AUCEvaluator

evaluator = AUCEvaluator()
all_scene_metrics = []

for scene_dir in Path("<benchmark_root>").glob("*/"):
    gt_loader = BSSLoader(scene_dir / "gt")
    pred_loader = BSSLoader(scene_dir / "pred")
    all_scene_metrics.append(evaluator.evaluate(gt_loader, pred_loader))

micro_result = AUCEvaluator.aggregate_micro(all_scene_metrics)
print("Micro-AUC @30°:", micro_result["AUC_30"])

```

This approach pools all errors globally, weighting the final trajectory AUC calculation by the number of valid pose pairs per scene.

### Macro Aggregation Across Scenes

```python
macro_result = AUCEvaluator.aggregate_macro(all_scene_metrics)
print("Macro-AUC @30°:", macro_result["AUC_30"])

```

Here, each scene’s AUC is computed independently then averaged, ensuring balanced representation across the dataset.

## Summary

- **Trajectory AUC calculation** integrates rotation and translation errors across all frame pairs to produce a robust SLAM benchmark metric.
- The pipeline executes in three stages: pose‑pair filtering, relative error computation, and cumulative distribution integration via `_calculate_auc_np`.
- **Micro aggregation** concatenates errors globally, weighting results by scene size.
- **Macro aggregation** averages per‑scene AUCs, giving equal importance to every test sequence.
- Both methods are implemented as static methods in `AUCEvaluator` within [`benchmark/benchmark/evaluation/auc.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/evaluation/auc.py).

## Frequently Asked Questions

### What is the difference between Racc and Tacc in the trajectory AUC calculation?

**Racc** measures the percentage of frame pairs where the relative rotation error falls below a threshold (e.g., 5°), while **Tacc** measures the same for translation angle errors. These accuracy metrics complement the AUC score by providing interpretable pass rates at specific precision thresholds.

### Why would I choose macro aggregation over micro aggregation for my SLAM benchmark?

Choose **macro aggregation** when your dataset contains scenes of vastly different lengths and you want to prevent long trajectories from skewing the results. This ensures that a 100‑frame office sequence and a 10,000‑frame outdoor drive contribute equally to the final score, providing a balanced view of system performance across diverse scenarios.

### Where does the lingbot‑map repository define the default angle thresholds?

The default thresholds `[3, 5, 15, 30]` (in degrees) are defined in the `AUCEvaluator` class constructor within [`benchmark/benchmark/evaluation/auc.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/evaluation/auc.py). These values are passed to `_calculate_auc_np` during both single‑scene evaluation and batch aggregation.

### How does the repository handle invalid or missing pose pairs?

The `_filter_valid_pose_pairs` function in [`benchmark/benchmark/evaluation/trajectory.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/evaluation/trajectory.py) aligns ground‑truth and prediction timestamps, removing any frames without corresponding matches. This ensures that the trajectory AUC calculation only considers valid relative transformations, preventing null errors from corrupting the metric.