How to Calculate Trajectory Accuracy on KITTI and Oxford Spires Using the LingBot-Map Benchmark

The LingBot-Map benchmark evaluates trajectory accuracy by computing Absolute Trajectory Error (ATE) and Relative Pose Error (RPE) using the evo library, aligning predictions to ground truth with a Sim(3) transform in benchmark/benchmark/evaluation/trajectory.py.

The Robbyant/lingbot-map repository provides a standardized evaluation framework for SLAM methods on the KITTI and Oxford Spires datasets. Understanding how to calculate trajectory accuracy on KITTI and Oxford Spires is essential for comparing camera pose estimation algorithms fairly. The evaluation pipeline leverages the evo library to compute industry-standard metrics while handling dataset-specific nuances like scale ambiguity.

Overview of the Trajectory Evaluation Pipeline

The heavy lifting occurs in benchmark/benchmark/evaluation/trajectory.py, which wraps the evo library to calculate ATE and RPE. The pipeline supports both KITTI and Oxford Spires through dataset-specific loaders that feed into a unified evaluation class.

Core Evaluation Architecture

All pose comparisons are handled by the TrajectoryEvaluator class, which orchestrates data loading, preprocessing, alignment, and metric computation. The evaluator accepts ground-truth and predicted trajectories as BSSLoader objects, ensuring consistent handling of SE(3) pose matrices across different datasets.

Step-by-Step Process to Calculate Trajectory Accuracy on KITTI and Oxford Spires

Loading Ground-Truth and Predicted Poses

Each dataset class (benchmark/datasets/kitti.py and benchmark/datasets/oxford_spires.py) implements a BSSLoader that reads pose files (*.txt) and returns (N, 4, 4) arrays of SE(3) matrices. The benchmark's run.py orchestrates the creation of GT and prediction loaders, passing them to the evaluator for processing.

Frame Matching and Validation

The TrajectoryEvaluator.evaluate method (lines 47-52) extracts ground-truth poses at indices matching the prediction frames using pred_loader.get_frame_indices(). This guarantees a one-to-one correspondence even when the SLAM method processes only a subset of frames.

Invalid poses containing NaN or Inf values are removed via _filter_valid_pose_pairs (lines 23-49). The function emits a warning when frames are dropped, ensuring robust evaluation even with sparse or corrupted outputs.

Orthogonalization and EVO Conversion

Before conversion to evo objects, rotation matrices undergo orthogonalization via SVD in _orthogonalize_se3 (lines 52-80) to satisfy SO(3) constraints. The _array_to_evo_trajectory function (lines 83-96) then converts the arrays to evo.core.trajectory.PoseTrajectory3D objects, preparing them for metric computation.

Sim(3) Alignment for Scale Correction

When align=True (default), the predicted trajectory is aligned to ground truth using a similarity transform (Sim(3)). The alignment block (lines 66-78) returns rotation R, translation t, and scale s, stored as a 4×4 matrix (traj_transform). This scale correction is critical for KITTI, where monocular methods produce arbitrary-scale outputs compared to metric ground truth.

Computing ATE and RPE Metrics

The evaluator calculates three standard metrics:

1. ATE: Calls evo.main_ape.ape with pose_relation=PoseRelation.translation_part and align=False (since alignment was already performed). Returns the RMSE of absolute translation error (lines 80-89).

2. RPE (Rotation): Calls evo.main_rpe.rpe with pose_relation=PoseRelation.rotation_angle_deg to compute the RMSE of rotational drift in degrees (lines 90-102).

3. RPE (Translation): Calls evo.main_rpe.rpe with pose_relation=PoseRelation.translation_part to compute the RMSE of relative translation error (lines 104-116).

The method returns a dictionary {'ate': ..., 'rpe_trans': ..., 'rpe_rot': ..., 'traj_transform': ...} to the benchmark driver (lines 119-124).

Handling KITTI vs. Oxford Spires Differences

Both datasets supply timestamped 4×4 pose matrices, but KITTI requires explicit scale correction via correct_scale=True (default) because its ground truth is metric while many SLAM methods produce scale-ambiguous trajectories. Oxford Spires provides scale-consistent trajectories, allowing the same pipeline to work without modification.

Practical Implementation Example

# Evaluate KITTI trajectory accuracy

from benchmark.benchmark.io.trajectory import BSSLoader
from benchmark.benchmark.evaluation.trajectory import TrajectoryEvaluator

gt_loader = BSSLoader("<path-to-kitti-gt>/poses.txt")
pred_loader = BSSLoader("<path-to-method-output>/poses.txt")

evaluator = TrajectoryEvaluator(align=True, correct_scale=True)
metrics = evaluator.evaluate(gt_loader, pred_loader)

print("ATE (m):", metrics["ate"])
print("RPE translation (m):", metrics["rpe_trans"])
print("RPE rotation (°):", metrics["rpe_rot"])

# Evaluate Oxford Spires trajectory accuracy

gt_loader = BSSLoader("<oxford-spire-gt>/poses.txt")
pred_loader = BSSLoader("<method-output>/poses.txt")

evaluator = TrajectoryEvaluator()  # defaults: align=True, correct_scale=True

metrics = evaluator.evaluate(gt_loader, pred_loader)

Summary

• The evaluation centers in benchmark/benchmark/evaluation/trajectory.py using the evo library to compute ATE and RPE.
• Sim(3) alignment corrects for scale differences between predicted and ground-truth trajectories, particularly critical for KITTI.
• The BSSLoader class standardizes pose loading across KITTI (benchmark/datasets/kitti.py) and Oxford Spires (benchmark/datasets/oxford_spires.py).
• Invalid poses are automatically filtered via _filter_valid_pose_pairs before metric computation.
• Rotation matrices undergo SVD orthogonalization in _orthogonalize_se3 to ensure valid SO(3) representations before evo processing.

Frequently Asked Questions

What metrics does the LingBot-Map benchmark use for trajectory evaluation?

The benchmark computes Absolute Trajectory Error (ATE) and Relative Pose Error (RPE). ATE measures the root-mean-square error of absolute translation differences after Sim(3) alignment, while RPE quantifies both rotational and translational drift between consecutive frames using the evo library.

Why is Sim(3) alignment necessary for KITTI trajectory evaluation?

Sim(3) alignment corrects for scale ambiguity between predicted trajectories and metric ground truth. KITTI provides poses in metric units, but monocular SLAM methods often output trajectories at arbitrary scales. The similarity transform estimates rotation, translation, and scale simultaneously in benchmark/benchmark/evaluation/trajectory.py, enabling fair comparison across methods with different scale conventions.

How does the benchmark handle missing or invalid poses?

The _filter_valid_pose_pairs function in trajectory.py (lines 23-49) automatically removes any pose pairs containing NaN or Inf values. When frames are dropped, the evaluator emits a warning but continues processing the valid subset, ensuring robust evaluation even with sparse SLAM outputs that only cover a fraction of the dataset frames.

Where is the evaluation logic implemented in the repository?

All trajectory evaluation logic resides in benchmark/benchmark/evaluation/trajectory.py. Dataset-specific loading is handled by benchmark/datasets/kitti.py and benchmark/datasets/oxford_spires.py, while the BSSLoader utility class is defined in benchmark/benchmark/io/trajectory.py. The orchestration occurs in benchmark/run.py, which creates loaders and invokes the TrajectoryEvaluator.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}