How Point Cloud Evaluation Computes Accuracy, Completeness, and Chamfer Distance

Point cloud evaluation in the lingbot-map repository calculates accuracy as the mean nearest-neighbor distance from predicted to ground-truth points, completeness as the reverse directional mean, and Chamfer distance as the average of both metrics, leveraging scipy's cKDTree for efficient spatial indexing.

The lingbot-map benchmark suite provides robust metrics for comparing reconstructed point clouds against ground truth. Understanding how it calculates accuracy, completeness, and Chamfer distance reveals the underlying geometric similarity measures that drive 3D reconstruction evaluation according to the Robbyant/lingbot-map source code.

The Three Core Metrics

The evaluation system in benchmark/benchmark/evaluation/points.py implements three distinct metrics that capture different aspects of geometric alignment between point clouds.

What Accuracy Measures

Accuracy quantifies how close the predicted points are to the ground truth. The implementation extracts XYZ coordinates from the first three columns of the input arrays, builds a KD-tree from the target points, and queries the distance from each source point to its nearest neighbor in the target.

In accuracy() (lines 54-78), the function calls distance(pred, gt) to compute the mean of all source-to-target distances. A lower value indicates that the reconstructed points lie closer to the actual surface.

What Completeness Measures

Completeness evaluates whether the prediction covers the entire ground-truth surface. Instead of measuring how far predictions are from truth, it measures how far ground-truth points are from the nearest predicted point.

The completeness() function (lines 80-102) achieves this by swapping the arguments—calling distance(gt, pred)—to compute the mean distance from target to source. High completeness indicates the reconstruction captures the full geometry without missing regions.

Chamfer Distance as the Symmetric Metric

Chamfer distance provides a single scalar that treats both directions equally. The chamfer_distance() function (lines 105-110) computes the arithmetic mean of accuracy and completeness: (accuracy + completeness) / 2.0.

This symmetric metric penalizes both geometric inaccuracy (predicted points far from truth) and incomplete coverage (missing parts of the ground truth) in a single value.

Implementation Details

The evaluation pipeline relies on efficient spatial data structures and directional distance calculations.

Nearest-Neighbor Search with KD-Trees

The foundation of all metrics lies in the distance() function (lines 10-40). The implementation uses scipy.spatial.cKDTree to accelerate spatial queries:

1. Extract only the XYZ coordinates: source_points[:, :3] and target_points[:, :3]
2. Instantiate the tree: tree = KDTree(target_xyz)
3. Query with parallel processing: tree.query(source_xyz, workers=-1)

This returns the Euclidean distance from each source point to its closest neighbor in the target cloud.

Directional Distance Calculation

The distinction between accuracy and completeness emerges solely from argument order:

• Accuracy: distance(pred, gt) returns dist_s2t, then np.mean(dist_s2t)
• Completeness: distance(gt, pred) returns dist_t2s, then np.mean(dist_t2s)

Both metrics use identical KD-tree logic but measure opposite directional flows.

Using the Evaluation API

The repository provides multiple interfaces for computing these metrics, from low-level functions to high-level evaluators.

Direct Metric Functions

For quick calculations on NumPy arrays, import the individual functions:

import numpy as np
from benchmark.benchmark.evaluation.points import accuracy, completeness, chamfer_distance

# Dummy point clouds (Nx3)

gt   = np.random.rand(1000, 3)
pred = np.random.rand(1000, 3)

acc = accuracy(pred, gt)          # mean distance pred→gt

comp = completeness(pred, gt)     # mean distance gt→pred

cham = chamfer_distance(acc, comp)

print(f"Accuracy: {acc:.4f}")
print(f"Completeness: {comp:.4f}")
print(f"Chamfer: {cham:.4f}")

High-Level Evaluation with evaluate_pointcloud

The evaluate_pointcloud() function (lines 250-390) handles reshaping, threshold-based precision/recall calculations, and returns a comprehensive metrics dictionary:

import numpy as np
from benchmark.benchmark.evaluation.points import evaluate_pointcloud

gt   = np.random.rand(2000, 3)
pred = np.random.rand(2000, 3)

metrics = evaluate_pointcloud(pred, gt)   # thresholds default to [0.01,0.02,0.05,0.10]

print("Chamfer:", metrics["chamfer"])
print("Accuracy:", metrics["accuracy"])
print("Completeness:", metrics["completeness"])

# threshold-specific values available as metrics["precision_0.05"]

The PointCloudEvaluator Class

For integration with the benchmark pipeline, the PointCloudEvaluator class (lines 425-470) wraps the evaluation logic:

from benchmark.benchmark.evaluation.points import PointCloudEvaluator

evaluator = PointCloudEvaluator(my_dataset)
metrics, gt_pts, pred_pts, thresholds = evaluator.evaluate(gt_loader, pred_loader)

print(metrics)          # dict with chamfer, accuracy, completeness, etc.

This class forwards dataset-specific calls to evaluate_pointcloud() and returns colored point clouds for visualization alongside the scalar metrics.

Summary

• Accuracy computes the mean nearest-neighbor distance from predicted points to ground truth, measuring geometric precision.
• Completeness reverses the direction, computing the mean distance from ground truth to predictions, measuring surface coverage.
• Chamfer distance averages accuracy and completeness to provide a symmetric metric that penalizes both deviation and missing regions.
• The implementation in benchmark/benchmark/evaluation/points.py uses scipy.spatial.cKDTree with workers=-1 for parallel nearest-neighbor queries.
• Three API levels exist: direct functions (accuracy(), completeness()), the convenience wrapper evaluate_pointcloud(), and the pipeline-ready PointCloudEvaluator class.

Frequently Asked Questions

What is the difference between accuracy and completeness in point cloud evaluation?

Accuracy measures how well the predicted points align with the ground-truth surface by calculating the mean distance from each predicted point to its nearest neighbor in the ground truth. Completeness measures coverage by calculating the mean distance from each ground-truth point to its nearest neighbor in the prediction. High accuracy means predicted points are close to the true surface, while high completeness means the reconstruction captures the full extent of the ground truth without gaps.

How does the Chamfer distance calculation differ from accuracy and completeness?

While accuracy and completeness are directional metrics (one-way distances), Chamfer distance is symmetric. The chamfer_distance() function computes the arithmetic mean of the accuracy and completeness values: (accuracy + completeness) / 2.0. This produces a single scalar that equally penalizes both geometric deviation (inaccuracy) and missing surface regions (incompleteness), making it suitable for overall reconstruction quality assessment.

Why does the implementation use scipy's cKDTree for distance calculations?

The distance() function utilizes scipy.spatial.cKDTree because KD-trees provide O(log n) query complexity for nearest-neighbor searches in 3D space, dramatically outperforming brute-force approaches. The implementation further optimizes performance by setting workers=-1 during the tree.query() call, enabling parallel processing across all available CPU cores. This allows efficient evaluation of point clouds containing thousands of points.

How can I evaluate my own point clouds using the lingbot-map benchmark?

Import the evaluation functions from benchmark.benchmark.evaluation.points and ensure your point clouds are NumPy arrays with shape (N, 3) containing XYZ coordinates. For simple metrics, use accuracy(pred, gt) and completeness(pred, gt). For comprehensive evaluation including threshold-based precision and recall, use evaluate_pointcloud(pred, gt, thresholds=[0.01, 0.05]). If integrating with the full benchmark pipeline, instantiate PointCloudEvaluator with a dataset object that implements the required loader interface.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}