Evaluation Metrics in LingBot-Map: A Complete Guide to Trajectory, Depth, Point-Cloud, and Pose-AUC Computation

LingBot-Map supports four categories of evaluation metrics—trajectory (ATE/RPE), depth-map (abs_rel, RMSE, δ thresholds), point-cloud (Chamfer distance, precision/recall), and pose-AUC—each computed using validated numpy and scipy routines in the benchmark/benchmark/evaluation/ module.

LingBot-Map is an open-source SLAM benchmarking framework that provides standardized evaluation utilities for camera trajectories, depth maps, point clouds, and pose accuracy. The evaluation metrics are implemented under benchmark/benchmark/evaluation/ and cover four distinct modalities, enabling comprehensive assessment of mapping and localization performance against ground-truth data.

Trajectory Evaluation Metrics

The TrajectoryEvaluator class in benchmark/benchmark/evaluation/trajectory.py computes standard SLAM trajectory errors using the EVO library backend. It processes predicted and ground-truth pose sequences through alignment, filtering, and orthogonalization before metric calculation.

Absolute Trajectory Error (ATE)

ATE measures the global consistency of the estimated trajectory by computing the root-mean-square error of translational deviations after Sim(3) alignment. The implementation calls main_ape.ape() with pose_relation=PoseRelation.translation_part and extracts the RMSE statistic from ape_result.stats["rmse"] (lines 78–89). The alignment step computes a 4×4 Sim(3) transformation matrix via traj_est_aligned.align() to handle scale, rotation, and translation differences between coordinate frames (lines 70–78).

Relative Pose Error (RPE)

RPE evaluates local consistency by measuring frame-to-frame motion errors. The evaluator computes two variants:

• RPE-trans: Translation RMSE using main_rpe.rpe(..., pose_relation=PoseRelation.translation_part, ...) (lines 90–99).
• RPE-rot: Rotation RMSE in degrees using main_rpe.rpe(..., pose_relation=PoseRelation.rotation_angle_deg, ...) (lines 90–99).

Both metrics compare relative transformations between successive poses rather than absolute world coordinates.

Pre-processing Pipeline

Before metric computation, the trajectory evaluator performs mandatory pre-processing:

1. Filtering: Removes NaN and Inf pose pairs via _filter_valid_pose_pairs.
2. Orthogonalization: Enforces valid SE(3) constraints on rotations using _orthogonalize_se3.
3. Format conversion: Transforms data into EVO’s PoseTrajectory3D objects.
4. Optional alignment: Applies Sim(3) alignment when align=True and correct_scale=True.

Depth-Map Evaluation Metrics

The DepthEvaluator in benchmark/benchmark/evaluation/depth.py computes pixel-wise depth accuracy metrics. All calculations occur in the _compute_metrics function (lines 9–30) after optional alignment and masking of invalid pixels.

Relative Error Metrics

• abs_rel (Absolute Relative Error): np.mean(np.abs(pred - gt) / gt) (lines 11–12).
• sq_rel (Squared Relative Error): np.mean((pred - gt) ** 2 / gt) (lines 12–13).

RMSE and Log-RMSE

• rmse: np.sqrt(np.mean((pred - gt) ** 2)) (lines 13–14).
• log_rmse: Root-mean-square error computed on logarithmic values, np.sqrt(np.mean((log(pred) - log(gt)) ** 2)), with clipping to handle numerical stability (lines 15–16).

Threshold Accuracy (δ)

The evaluator reports the percentage of pixels where the ratio between prediction and ground truth falls below successive thresholds:

• δ1.25: Error ratio < 1.25.
• δ1.5625: Error ratio < 1.25².
• δ1.953125: Error ratio < 1.25³.

Computation uses np.mean((max_ratio < threshold).astype(float)) * 100, where max_ratio = max(pred/gt, gt/pred) (lines 17–20).

Alignment and Masking

Ground-truth depth is clipped to a valid range (gt_clip), and invalid pixels (NaN, Inf, zero depth, or outside range) are excluded from computation. The evaluator supports three alignment modes: scale_only, lstsq, and none.

Point-Cloud Evaluation Metrics

The evaluate_pointcloud function in benchmark/benchmark/evaluation/points.py (lines 249–340) provides a unified interface for 3D geometry evaluation using KD-tree nearest-neighbor searches.

Accuracy and Completeness

• Accuracy: Mean nearest-neighbor distance from source (predicted) to target (ground-truth) points, computed via np.mean on KD-tree distances (lines 54–57).
• Completeness: Mean nearest-neighbor distance from target to source (reverse direction), indicating ground-truth coverage (lines 80–84).

Chamfer Distance

Chamfer Distance measures the symmetric reconstruction error as the average of accuracy and completeness: (accuracy + completeness) / 2 (lines 105–108).

Precision, Recall, and F1-Score

For configurable distance thresholds (default: 0.01, 0.05, 0.10):

• Precision @ τ: Percentage of source points within distance τ of any target point, np.mean(dist_s2t < τ) * 100 (lines 135–164).
• Recall @ τ: Percentage of target points within distance τ of any source point, computed similarly with dist_t2s (lines 168–190).
• F1-score @ τ: Harmonic mean of precision and recall, 2 * (prec * rec) / (prec + rec) (lines 207–230).

The implementation uses scipy.spatial.cKDTree for O(log N) nearest-neighbor queries, pre-computing distance matrices once for efficiency when calculating multiple metrics.

Pose-AUC Evaluation Metrics

The AUCEvaluator in benchmark/benchmark/evaluation/auc.py evaluates relative pose error distributions across all frame pairs, reporting accuracy percentages and area-under-curve metrics.

Rotation and Translation Accuracy

• Racc_θ: Percentage of relative rotation errors below threshold θ, computed as np.mean(rel_rangle < θ) * 100 (lines 96–98).
• Tacc_θ: Percentage of relative translation-angle errors below threshold θ, computed as np.mean(rel_tangle < θ) * 100 (lines 96–99).

Area Under the Curve (AUC)

AUC_θ represents the area under the cumulative distribution curve of max(rotation_error, translation_error) up to threshold θ. The implementation builds a normalized histogram of maximum errors and computes np.mean(np.cumsum(normalized_histogram)) (lines 172–181).

Default evaluation thresholds are 3°, 5°, 15°, and 30°. The evaluator first converts poses to world-to-camera format, aligns both sequences to the first camera frame, then computes all pairwise relative pose errors via _se3_to_relative_pose_error (lines 57–69).

Practical Usage Examples

Evaluating Trajectory Accuracy

from benchmark.benchmark.evaluation.trajectory import TrajectoryEvaluator
from benchmark.benchmark.core.loader import BSSLoader

gt_loader = BSSLoader('/path/to/gt')
pred_loader = BSSLoader('/path/to/pred')

evaluator = TrajectoryEvaluator(align=True, correct_scale=True)
metrics = evaluator.evaluate(gt_loader, pred_loader)

print(metrics)

# Output: {'ate': 0.045, 'rpe_trans': 0.012, 'rpe_rot': 0.89, 'traj_transform': [...]}

Computing Depth Metrics

from benchmark.benchmark.evaluation.depth import DepthEvaluator
from benchmark.benchmark.core.loader import BSSLoader

gt_loader = BSSLoader('/path/to/gt')
pred_loader = BSSLoader('/path/to/pred')

evaluator = DepthEvaluator(align='scale_only')
metrics = evaluator.evaluate(gt_loader, pred_loader)

print(metrics)

# Output includes: abs_rel, sq_rel, rmse, log_rmse, delta_1_25, etc.

Measuring Point-Cloud Quality

import numpy as np
from benchmark.benchmark.evaluation.points import evaluate_pointcloud

pred_pts = np.random.rand(1000, 3)  # Predicted point cloud

gt_pts = np.random.rand(1000, 3)    # Ground-truth

metrics = evaluate_pointcloud(pred_pts, gt_pts, thresholds=[0.01, 0.05, 0.10])

print(f"Chamfer: {metrics['chamfer']:.4f}")
print(f"Precision@0.05: {metrics['precision_0.05']:.2f}%")
print(f"F1@0.05: {metrics['f1_score_0.05']:.4f}")

Calculating Pose-AUC

from benchmark.benchmark.evaluation.auc import AUCEvaluator
from benchmark.benchmark.core.loader import BSSLoader

gt_loader = BSSLoader('/path/to/gt')
pred_loader = BSSLoader('/path/to/pred')

evaluator = AUCEvaluator()
metrics = evaluator.evaluate(gt_loader, pred_loader)

print(f"AUC@3°: {metrics['AUC_03']:.2f}")
print(f"Rotation Acc@3°: {metrics['Racc_03']:.2f}%")
print(f"Translation Acc@3°: {metrics['Tacc_03']:.2f}%")

Summary

• Trajectory evaluation provides ATE (global RMSE) and RPE (local frame-to-frame errors) with optional Sim(3) alignment via TrajectoryEvaluator.
• Depth evaluation computes abs_rel, sq_rel, RMSE, log-RMSE, and three δ-threshold accuracies with configurable alignment modes in DepthEvaluator.
• Point-cloud evaluation returns Chamfer distance, accuracy, completeness, and threshold-based precision/recall/F1 using KD-tree nearest-neighbor lookups.
• Pose-AUC evaluation calculates rotation/translation accuracy percentages and cumulative AUC scores at standard thresholds (3°, 5°, 15°, 30°).

Frequently Asked Questions

What is the difference between ATE and RPE in LingBot-Map?

Absolute Trajectory Error (ATE) measures global drift by comparing the entire aligned trajectory to ground truth, reporting the RMSE of translation errors. Relative Pose Error (RPE) measures local consistency by computing frame-to-frame motion errors, providing separate metrics for translation (meters) and rotation (degrees). ATE is sensitive to global alignment quality, while RPE reveals local tracking stability.

How does LingBot-Map handle depth-map alignment before computing metrics?

The DepthEvaluator supports three alignment strategies specified via the align parameter: scale_only applies a global scale factor, lstsq performs least-squares fitting for scale and offset, and none disables alignment. Ground-truth values are clipped to valid ranges, and invalid pixels (NaN, Inf, or zero depth) are masked before metric computation in _compute_metrics.

What threshold values are used for Pose-AUC evaluation?

The AUCEvaluator uses default thresholds of 3°, 5°, 15°, and 30° for both rotation and translation accuracy metrics. These values represent common angular tolerances in SLAM evaluation, with the Area Under Curve (AUC) computed as the normalized cumulative distribution of maximum errors up to each threshold.

Which pre-processing steps are applied to trajectories before metric computation?

Trajectories undergo four pre-processing steps in TrajectoryEvaluator.evaluate: (1) filtering of NaN/Inf poses via _filter_valid_pose_pairs, (2) SE(3) orthogonalization via _orthogonalize_se3 to enforce valid rotation matrices, (3) conversion to EVO PoseTrajectory3D format, and (4) optional Sim(3) alignment to handle scale ambiguities between prediction and ground truth.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}