# How to Calculate Trajectory Accuracy on KITTI and Oxford Spires Using the LingBot-Map Benchmark

> Learn how the LingBot-Map benchmark calculates trajectory accuracy on KITTI and Oxford Spires using ATE and RPE metrics with the evo library and Sim(3) alignment.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: how-to-guide
- Published: 2026-07-04

---

**The LingBot-Map benchmark evaluates trajectory accuracy by computing Absolute Trajectory Error (ATE) and Relative Pose Error (RPE) using the evo library, aligning predictions to ground truth with a Sim(3) transform in [`benchmark/benchmark/evaluation/trajectory.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/evaluation/trajectory.py).**

The Robbyant/lingbot-map repository provides a standardized evaluation framework for SLAM methods on the KITTI and Oxford Spires datasets. Understanding how to calculate trajectory accuracy on KITTI and Oxford Spires is essential for comparing camera pose estimation algorithms fairly. The evaluation pipeline leverages the evo library to compute industry-standard metrics while handling dataset-specific nuances like scale ambiguity.

## Overview of the Trajectory Evaluation Pipeline

The heavy lifting occurs in [`benchmark/benchmark/evaluation/trajectory.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/evaluation/trajectory.py), which wraps the evo library to calculate ATE and RPE. The pipeline supports both KITTI and Oxford Spires through dataset-specific loaders that feed into a unified evaluation class.

### Core Evaluation Architecture

All pose comparisons are handled by the `TrajectoryEvaluator` class, which orchestrates data loading, preprocessing, alignment, and metric computation. The evaluator accepts **ground-truth** and **predicted** trajectories as `BSSLoader` objects, ensuring consistent handling of SE(3) pose matrices across different datasets.

## Step-by-Step Process to Calculate Trajectory Accuracy on KITTI and Oxford Spires

### Loading Ground-Truth and Predicted Poses

Each dataset class ([`benchmark/datasets/kitti.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/datasets/kitti.py) and [`benchmark/datasets/oxford_spires.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/datasets/oxford_spires.py)) implements a `BSSLoader` that reads pose files (`*.txt`) and returns `(N, 4, 4)` arrays of SE(3) matrices. The benchmark's [`run.py`](https://github.com/Robbyant/lingbot-map/blob/main/run.py) orchestrates the creation of GT and prediction loaders, passing them to the evaluator for processing.

### Frame Matching and Validation

The `TrajectoryEvaluator.evaluate` method (lines 47-52) extracts ground-truth poses at indices matching the prediction frames using `pred_loader.get_frame_indices()`. This guarantees a one-to-one correspondence even when the SLAM method processes only a subset of frames.

Invalid poses containing **NaN** or **Inf** values are removed via `_filter_valid_pose_pairs` (lines 23-49). The function emits a warning when frames are dropped, ensuring robust evaluation even with sparse or corrupted outputs.

### Orthogonalization and EVO Conversion

Before conversion to evo objects, rotation matrices undergo orthogonalization via SVD in `_orthogonalize_se3` (lines 52-80) to satisfy SO(3) constraints. The `_array_to_evo_trajectory` function (lines 83-96) then converts the arrays to `evo.core.trajectory.PoseTrajectory3D` objects, preparing them for metric computation.

### Sim(3) Alignment for Scale Correction

When `align=True` (default), the predicted trajectory is aligned to ground truth using a **similarity transform** (Sim(3)). The alignment block (lines 66-78) returns rotation `R`, translation `t`, and scale `s`, stored as a 4×4 matrix (`traj_transform`). This scale correction is critical for KITTI, where monocular methods produce arbitrary-scale outputs compared to metric ground truth.

### Computing ATE and RPE Metrics

The evaluator calculates three standard metrics:

1. **ATE**: Calls `evo.main_ape.ape` with `pose_relation=PoseRelation.translation_part` and `align=False` (since alignment was already performed). Returns the RMSE of absolute translation error (lines 80-89).

2. **RPE (Rotation)**: Calls `evo.main_rpe.rpe` with `pose_relation=PoseRelation.rotation_angle_deg` to compute the RMSE of rotational drift in degrees (lines 90-102).

3. **RPE (Translation)**: Calls `evo.main_rpe.rpe` with `pose_relation=PoseRelation.translation_part` to compute the RMSE of relative translation error (lines 104-116).

The method returns a dictionary `{'ate': ..., 'rpe_trans': ..., 'rpe_rot': ..., 'traj_transform': ...}` to the benchmark driver (lines 119-124).

## Handling KITTI vs. Oxford Spires Differences

Both datasets supply timestamped 4×4 pose matrices, but KITTI requires explicit scale correction via `correct_scale=True` (default) because its ground truth is metric while many SLAM methods produce scale-ambiguous trajectories. Oxford Spires provides scale-consistent trajectories, allowing the same pipeline to work without modification.

## Practical Implementation Example

```python

# Evaluate KITTI trajectory accuracy

from benchmark.benchmark.io.trajectory import BSSLoader
from benchmark.benchmark.evaluation.trajectory import TrajectoryEvaluator

gt_loader = BSSLoader("<path-to-kitti-gt>/poses.txt")
pred_loader = BSSLoader("<path-to-method-output>/poses.txt")

evaluator = TrajectoryEvaluator(align=True, correct_scale=True)
metrics = evaluator.evaluate(gt_loader, pred_loader)

print("ATE (m):", metrics["ate"])
print("RPE translation (m):", metrics["rpe_trans"])
print("RPE rotation (°):", metrics["rpe_rot"])

```

```python

# Evaluate Oxford Spires trajectory accuracy

gt_loader = BSSLoader("<oxford-spire-gt>/poses.txt")
pred_loader = BSSLoader("<method-output>/poses.txt")

evaluator = TrajectoryEvaluator()  # defaults: align=True, correct_scale=True

metrics = evaluator.evaluate(gt_loader, pred_loader)

```

## Summary

- The evaluation centers in [`benchmark/benchmark/evaluation/trajectory.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/evaluation/trajectory.py) using the evo library to compute ATE and RPE.
- **Sim(3) alignment** corrects for scale differences between predicted and ground-truth trajectories, particularly critical for KITTI.
- The `BSSLoader` class standardizes pose loading across KITTI ([`benchmark/datasets/kitti.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/datasets/kitti.py)) and Oxford Spires ([`benchmark/datasets/oxford_spires.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/datasets/oxford_spires.py)).
- Invalid poses are automatically filtered via `_filter_valid_pose_pairs` before metric computation.
- Rotation matrices undergo SVD orthogonalization in `_orthogonalize_se3` to ensure valid SO(3) representations before evo processing.

## Frequently Asked Questions

### What metrics does the LingBot-Map benchmark use for trajectory evaluation?

The benchmark computes **Absolute Trajectory Error (ATE)** and **Relative Pose Error (RPE)**. ATE measures the root-mean-square error of absolute translation differences after Sim(3) alignment, while RPE quantifies both rotational and translational drift between consecutive frames using the evo library.

### Why is Sim(3) alignment necessary for KITTI trajectory evaluation?

Sim(3) alignment corrects for scale ambiguity between predicted trajectories and metric ground truth. KITTI provides poses in metric units, but monocular SLAM methods often output trajectories at arbitrary scales. The similarity transform estimates rotation, translation, and scale simultaneously in [`benchmark/benchmark/evaluation/trajectory.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/evaluation/trajectory.py), enabling fair comparison across methods with different scale conventions.

### How does the benchmark handle missing or invalid poses?

The `_filter_valid_pose_pairs` function in [`trajectory.py`](https://github.com/Robbyant/lingbot-map/blob/main/trajectory.py) (lines 23-49) automatically removes any pose pairs containing NaN or Inf values. When frames are dropped, the evaluator emits a warning but continues processing the valid subset, ensuring robust evaluation even with sparse SLAM outputs that only cover a fraction of the dataset frames.

### Where is the evaluation logic implemented in the repository?

All trajectory evaluation logic resides in [`benchmark/benchmark/evaluation/trajectory.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/evaluation/trajectory.py). Dataset-specific loading is handled by [`benchmark/datasets/kitti.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/datasets/kitti.py) and [`benchmark/datasets/oxford_spires.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/datasets/oxford_spires.py), while the `BSSLoader` utility class is defined in [`benchmark/benchmark/io/trajectory.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/io/trajectory.py). The orchestration occurs in [`benchmark/run.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/run.py), which creates loaders and invokes the `TrajectoryEvaluator`.