# Evaluation Metrics in LingBot-Map: A Complete Guide to Trajectory, Depth, Point-Cloud, and Pose-AUC Computation

> Explore LingBot-Map evaluation metrics for trajectory, depth, point-cloud, and pose-AUC. Learn how ATE, RPE, Chamfer distance, and more are computed with numpy and scipy.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: deep-dive
- Published: 2026-07-02

---

**LingBot-Map supports four categories of evaluation metrics—trajectory (ATE/RPE), depth-map (abs_rel, RMSE, δ thresholds), point-cloud (Chamfer distance, precision/recall), and pose-AUC—each computed using validated numpy and scipy routines in the `benchmark/benchmark/evaluation/` module.**

LingBot-Map is an open-source SLAM benchmarking framework that provides standardized evaluation utilities for camera trajectories, depth maps, point clouds, and pose accuracy. The evaluation metrics are implemented under `benchmark/benchmark/evaluation/` and cover four distinct modalities, enabling comprehensive assessment of mapping and localization performance against ground-truth data.

## Trajectory Evaluation Metrics

The `TrajectoryEvaluator` class in [`benchmark/benchmark/evaluation/trajectory.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/evaluation/trajectory.py) computes standard SLAM trajectory errors using the EVO library backend. It processes predicted and ground-truth pose sequences through alignment, filtering, and orthogonalization before metric calculation.

### Absolute Trajectory Error (ATE)

**ATE** measures the global consistency of the estimated trajectory by computing the root-mean-square error of translational deviations after Sim(3) alignment. The implementation calls `main_ape.ape()` with `pose_relation=PoseRelation.translation_part` and extracts the RMSE statistic from `ape_result.stats["rmse"]` (lines 78–89). The alignment step computes a 4×4 Sim(3) transformation matrix via `traj_est_aligned.align()` to handle scale, rotation, and translation differences between coordinate frames (lines 70–78).

### Relative Pose Error (RPE)

**RPE** evaluates local consistency by measuring frame-to-frame motion errors. The evaluator computes two variants:

- **RPE-trans**: Translation RMSE using `main_rpe.rpe(..., pose_relation=PoseRelation.translation_part, ...)` (lines 90–99).
- **RPE-rot**: Rotation RMSE in degrees using `main_rpe.rpe(..., pose_relation=PoseRelation.rotation_angle_deg, ...)` (lines 90–99).

Both metrics compare relative transformations between successive poses rather than absolute world coordinates.

### Pre-processing Pipeline

Before metric computation, the trajectory evaluator performs mandatory pre-processing:

1. **Filtering**: Removes NaN and Inf pose pairs via `_filter_valid_pose_pairs`.
2. **Orthogonalization**: Enforces valid SE(3) constraints on rotations using `_orthogonalize_se3`.
3. **Format conversion**: Transforms data into EVO’s `PoseTrajectory3D` objects.
4. **Optional alignment**: Applies Sim(3) alignment when `align=True` and `correct_scale=True`.

## Depth-Map Evaluation Metrics

The `DepthEvaluator` in [`benchmark/benchmark/evaluation/depth.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/evaluation/depth.py) computes pixel-wise depth accuracy metrics. All calculations occur in the `_compute_metrics` function (lines 9–30) after optional alignment and masking of invalid pixels.

### Relative Error Metrics

- **abs_rel** (Absolute Relative Error): `np.mean(np.abs(pred - gt) / gt)` (lines 11–12).
- **sq_rel** (Squared Relative Error): `np.mean((pred - gt) ** 2 / gt)` (lines 12–13).

### RMSE and Log-RMSE

- **rmse**: `np.sqrt(np.mean((pred - gt) ** 2))` (lines 13–14).
- **log_rmse**: Root-mean-square error computed on logarithmic values, `np.sqrt(np.mean((log(pred) - log(gt)) ** 2))`, with clipping to handle numerical stability (lines 15–16).

### Threshold Accuracy (δ)

The evaluator reports the percentage of pixels where the ratio between prediction and ground truth falls below successive thresholds:

- **δ<sub>1.25</sub>**: Error ratio `< 1.25`.
- **δ<sub>1.5625</sub>**: Error ratio `< 1.25²`.
- **δ<sub>1.953125</sub>**: Error ratio `< 1.25³`.

Computation uses `np.mean((max_ratio < threshold).astype(float)) * 100`, where `max_ratio = max(pred/gt, gt/pred)` (lines 17–20).

### Alignment and Masking

Ground-truth depth is clipped to a valid range (`gt_clip`), and invalid pixels (NaN, Inf, zero depth, or outside range) are excluded from computation. The evaluator supports three alignment modes: `scale_only`, `lstsq`, and `none`.

## Point-Cloud Evaluation Metrics

The `evaluate_pointcloud` function in [`benchmark/benchmark/evaluation/points.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/evaluation/points.py) (lines 249–340) provides a unified interface for 3D geometry evaluation using KD-tree nearest-neighbor searches.

### Accuracy and Completeness

- **Accuracy**: Mean nearest-neighbor distance from source (predicted) to target (ground-truth) points, computed via `np.mean` on KD-tree distances (lines 54–57).
- **Completeness**: Mean nearest-neighbor distance from target to source (reverse direction), indicating ground-truth coverage (lines 80–84).

### Chamfer Distance

**Chamfer Distance** measures the symmetric reconstruction error as the average of accuracy and completeness: `(accuracy + completeness) / 2` (lines 105–108).

### Precision, Recall, and F1-Score

For configurable distance thresholds (default: 0.01, 0.05, 0.10):

- **Precision @ τ**: Percentage of source points within distance τ of any target point, `np.mean(dist_s2t < τ) * 100` (lines 135–164).
- **Recall @ τ**: Percentage of target points within distance τ of any source point, computed similarly with `dist_t2s` (lines 168–190).
- **F1-score @ τ**: Harmonic mean of precision and recall, `2 * (prec * rec) / (prec + rec)` (lines 207–230).

The implementation uses `scipy.spatial.cKDTree` for O(log N) nearest-neighbor queries, pre-computing distance matrices once for efficiency when calculating multiple metrics.

## Pose-AUC Evaluation Metrics

The `AUCEvaluator` in [`benchmark/benchmark/evaluation/auc.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/evaluation/auc.py) evaluates relative pose error distributions across all frame pairs, reporting accuracy percentages and area-under-curve metrics.

### Rotation and Translation Accuracy

- **Racc_θ**: Percentage of relative rotation errors below threshold θ, computed as `np.mean(rel_rangle < θ) * 100` (lines 96–98).
- **Tacc_θ**: Percentage of relative translation-angle errors below threshold θ, computed as `np.mean(rel_tangle < θ) * 100` (lines 96–99).

### Area Under the Curve (AUC)

**AUC_θ** represents the area under the cumulative distribution curve of `max(rotation_error, translation_error)` up to threshold θ. The implementation builds a normalized histogram of maximum errors and computes `np.mean(np.cumsum(normalized_histogram))` (lines 172–181).

Default evaluation thresholds are **3°, 5°, 15°, and 30°**. The evaluator first converts poses to world-to-camera format, aligns both sequences to the first camera frame, then computes all pairwise relative pose errors via `_se3_to_relative_pose_error` (lines 57–69).

## Practical Usage Examples

### Evaluating Trajectory Accuracy

```python
from benchmark.benchmark.evaluation.trajectory import TrajectoryEvaluator
from benchmark.benchmark.core.loader import BSSLoader

gt_loader = BSSLoader('/path/to/gt')
pred_loader = BSSLoader('/path/to/pred')

evaluator = TrajectoryEvaluator(align=True, correct_scale=True)
metrics = evaluator.evaluate(gt_loader, pred_loader)

print(metrics)

# Output: {'ate': 0.045, 'rpe_trans': 0.012, 'rpe_rot': 0.89, 'traj_transform': [...]}

```

### Computing Depth Metrics

```python
from benchmark.benchmark.evaluation.depth import DepthEvaluator
from benchmark.benchmark.core.loader import BSSLoader

gt_loader = BSSLoader('/path/to/gt')
pred_loader = BSSLoader('/path/to/pred')

evaluator = DepthEvaluator(align='scale_only')
metrics = evaluator.evaluate(gt_loader, pred_loader)

print(metrics)

# Output includes: abs_rel, sq_rel, rmse, log_rmse, delta_1_25, etc.

```

### Measuring Point-Cloud Quality

```python
import numpy as np
from benchmark.benchmark.evaluation.points import evaluate_pointcloud

pred_pts = np.random.rand(1000, 3)  # Predicted point cloud

gt_pts = np.random.rand(1000, 3)    # Ground-truth

metrics = evaluate_pointcloud(pred_pts, gt_pts, thresholds=[0.01, 0.05, 0.10])

print(f"Chamfer: {metrics['chamfer']:.4f}")
print(f"Precision@0.05: {metrics['precision_0.05']:.2f}%")
print(f"F1@0.05: {metrics['f1_score_0.05']:.4f}")

```

### Calculating Pose-AUC

```python
from benchmark.benchmark.evaluation.auc import AUCEvaluator
from benchmark.benchmark.core.loader import BSSLoader

gt_loader = BSSLoader('/path/to/gt')
pred_loader = BSSLoader('/path/to/pred')

evaluator = AUCEvaluator()
metrics = evaluator.evaluate(gt_loader, pred_loader)

print(f"AUC@3°: {metrics['AUC_03']:.2f}")
print(f"Rotation Acc@3°: {metrics['Racc_03']:.2f}%")
print(f"Translation Acc@3°: {metrics['Tacc_03']:.2f}%")

```

## Summary

- **Trajectory evaluation** provides ATE (global RMSE) and RPE (local frame-to-frame errors) with optional Sim(3) alignment via `TrajectoryEvaluator`.
- **Depth evaluation** computes abs_rel, sq_rel, RMSE, log-RMSE, and three δ-threshold accuracies with configurable alignment modes in `DepthEvaluator`.
- **Point-cloud evaluation** returns Chamfer distance, accuracy, completeness, and threshold-based precision/recall/F1 using KD-tree nearest-neighbor lookups.
- **Pose-AUC evaluation** calculates rotation/translation accuracy percentages and cumulative AUC scores at standard thresholds (3°, 5°, 15°, 30°).

## Frequently Asked Questions

### What is the difference between ATE and RPE in LingBot-Map?

**Absolute Trajectory Error (ATE)** measures global drift by comparing the entire aligned trajectory to ground truth, reporting the RMSE of translation errors. **Relative Pose Error (RPE)** measures local consistency by computing frame-to-frame motion errors, providing separate metrics for translation (meters) and rotation (degrees). ATE is sensitive to global alignment quality, while RPE reveals local tracking stability.

### How does LingBot-Map handle depth-map alignment before computing metrics?

The `DepthEvaluator` supports three alignment strategies specified via the `align` parameter: `scale_only` applies a global scale factor, `lstsq` performs least-squares fitting for scale and offset, and `none` disables alignment. Ground-truth values are clipped to valid ranges, and invalid pixels (NaN, Inf, or zero depth) are masked before metric computation in `_compute_metrics`.

### What threshold values are used for Pose-AUC evaluation?

The `AUCEvaluator` uses default thresholds of **3°, 5°, 15°, and 30°** for both rotation and translation accuracy metrics. These values represent common angular tolerances in SLAM evaluation, with the Area Under Curve (AUC) computed as the normalized cumulative distribution of maximum errors up to each threshold.

### Which pre-processing steps are applied to trajectories before metric computation?

Trajectories undergo four pre-processing steps in `TrajectoryEvaluator.evaluate`: (1) filtering of NaN/Inf poses via `_filter_valid_pose_pairs`, (2) SE(3) orthogonalization via `_orthogonalize_se3` to enforce valid rotation matrices, (3) conversion to EVO `PoseTrajectory3D` format, and (4) optional Sim(3) alignment to handle scale ambiguities between prediction and ground truth.