How Trajectory AUC Calculation Works: Micro vs Macro Aggregation in Lingbot‑Map

Trajectory AUC calculation in lingbot‑map pools per‑scene rotation and translation errors to compute the area under the cumulative‑distribution curve, either globally via micro‑aggregation or as averaged per‑scene scores via macro‑aggregation.

The Robbyant/lingbot-map repository provides a comprehensive benchmarking toolkit for SLAM and visual odometry systems. At its core lies a trajectory AUC calculation that quantifies trajectory accuracy by measuring relative pose errors across all frame pairs. This metric supports two distinct aggregation strategies—micro and macro—that handle multi‑scene datasets differently depending on whether you want to weight scenes by their size or treat them equally.

The Three Stages of Trajectory AUC Calculation

The AUC pipeline implemented in benchmark/benchmark/evaluation/auc.py processes trajectories through three distinct logical phases before reaching the final metric.

Pose‑Pair Preparation

First, ground‑truth and predicted trajectories are loaded and aligned. The AUCEvaluator.evaluate method calls _filter_valid_pose_pairs (defined in benchmark/benchmark/evaluation/trajectory.py) to ensure only valid corresponding poses remain. The system transforms coordinates to the world‑to‑camera (W2C) convention using utilities from benchmark/benchmark/geometry/transform.py.

Relative‑Pose Error Computation

For every unordered pair of valid frames, the system computes relative transformation errors. The helper _build_pair_index generates all frame combinations, while _se3_to_relative_pose_error calculates:

• Rotation error via _rotation_angle
• Translation error via _translation_angle

These functions appear around lines 57–68 in the source, processing SE(3) transforms to produce angular deviations in degrees.

AUC and Accuracy Metrics

Using default angle thresholds of 3°, 5°, 15°, and 30°, the evaluator computes Racc (rotation accuracy) and Tacc (translation accuracy) as the fraction of pairs with errors below each threshold. The core trajectory AUC calculation happens in _calculate_auc_np (lines 71–81), which integrates the cumulative distribution of the maximum error (rotation or translation) across all pairs.

Micro vs Macro Aggregation

When benchmarking datasets containing multiple scenes, the repository offers two aggregation modes that answer different statistical questions.

Micro Aggregation: Weight by Pair Count

Micro aggregation pools all raw rotation (rError) and translation (tError) values from every scene into single massive arrays. It then computes one global AUC, Racc, and Tacc across this entire pool.

• Implementation: AUCEvaluator.aggregate_micro (lines 14–25 in auc.py)
• Behavior: Scenes with more frame pairs contribute proportionally more to the final score
• Use case: Prioritizing overall system performance where larger sequences naturally carry more weight

The method concatenates error arrays using np.concatenate before calling _calculate_auc_np, ensuring the error distribution reflects the entire dataset’s raw data.

Macro Aggregation: Equal Scene Weighting

Macro aggregation computes AUC, Racc, and Tacc independently for each scene, then averages these per‑scene scores using simple arithmetic mean.

• Implementation: AUCEvaluator.aggregate_macro (lines 27–35 in auc.py)
• Behavior: Every scene contributes equally regardless of frame count
• Use case: Fair comparison across benchmarks where short and long sequences should have equal influence

This approach prevents a single long trajectory from dominating the final metric, making it ideal for standardized leaderboards.

Implementation Details

Both aggregation strategies reuse the same low‑level math. The _calculate_auc_np function accepts error arrays and thresholds, returning the area under the curve and threshold‑accuracies. The key difference lies in input preparation:

• Micro: Concatenates rError and tError lists from all scenes before calculation
• Macro: Calls _calculate_auc_np per scene, stores the results, then applies np.mean across scenes

Default thresholds are defined as [3, 5, 15, 30] degrees, though these can be customized during AUCEvaluator instantiation.

Practical Code Examples

Evaluate a Single Scene

from benchmark.benchmark.io.trajectory import BSSLoader
from benchmark.benchmark.evaluation.auc import AUCEvaluator

gt_loader = BSSLoader("<gt_folder>")
pred_loader = BSSLoader("<pred_folder>")

evaluator = AUCEvaluator()  # Default thresholds: [3, 5, 15, 30]

scene_metrics = evaluator.evaluate(gt_loader, pred_loader)

print("Scene AUC @ 5°:", scene_metrics["AUC_05"])
print("Scene Racc @ 15°:", scene_metrics["Racc_15"])

The evaluate method returns a dictionary containing raw error arrays (rError, tError) that subsequent aggregation functions consume.

Micro Aggregation Across Scenes

from pathlib import Path
from benchmark.benchmark.evaluation.auc import AUCEvaluator

evaluator = AUCEvaluator()
all_scene_metrics = []

for scene_dir in Path("<benchmark_root>").glob("*/"):
    gt_loader = BSSLoader(scene_dir / "gt")
    pred_loader = BSSLoader(scene_dir / "pred")
    all_scene_metrics.append(evaluator.evaluate(gt_loader, pred_loader))

micro_result = AUCEvaluator.aggregate_micro(all_scene_metrics)
print("Micro-AUC @30°:", micro_result["AUC_30"])

This approach pools all errors globally, weighting the final trajectory AUC calculation by the number of valid pose pairs per scene.

Macro Aggregation Across Scenes

macro_result = AUCEvaluator.aggregate_macro(all_scene_metrics)
print("Macro-AUC @30°:", macro_result["AUC_30"])

Here, each scene’s AUC is computed independently then averaged, ensuring balanced representation across the dataset.

Summary

• Trajectory AUC calculation integrates rotation and translation errors across all frame pairs to produce a robust SLAM benchmark metric.
• The pipeline executes in three stages: pose‑pair filtering, relative error computation, and cumulative distribution integration via _calculate_auc_np.
• Micro aggregation concatenates errors globally, weighting results by scene size.
• Macro aggregation averages per‑scene AUCs, giving equal importance to every test sequence.
• Both methods are implemented as static methods in AUCEvaluator within benchmark/benchmark/evaluation/auc.py.

Frequently Asked Questions

What is the difference between Racc and Tacc in the trajectory AUC calculation?

Racc measures the percentage of frame pairs where the relative rotation error falls below a threshold (e.g., 5°), while Tacc measures the same for translation angle errors. These accuracy metrics complement the AUC score by providing interpretable pass rates at specific precision thresholds.

Why would I choose macro aggregation over micro aggregation for my SLAM benchmark?

Choose macro aggregation when your dataset contains scenes of vastly different lengths and you want to prevent long trajectories from skewing the results. This ensures that a 100‑frame office sequence and a 10,000‑frame outdoor drive contribute equally to the final score, providing a balanced view of system performance across diverse scenarios.

Where does the lingbot‑map repository define the default angle thresholds?

The default thresholds [3, 5, 15, 30] (in degrees) are defined in the AUCEvaluator class constructor within benchmark/benchmark/evaluation/auc.py. These values are passed to _calculate_auc_np during both single‑scene evaluation and batch aggregation.

How does the repository handle invalid or missing pose pairs?

The _filter_valid_pose_pairs function in benchmark/benchmark/evaluation/trajectory.py aligns ground‑truth and prediction timestamps, removing any frames without corresponding matches. This ensures that the trajectory AUC calculation only considers valid relative transformations, preventing null errors from corrupting the metric.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}