Why Custom Distance Functions Are Not Supported in pybind11-rdp: A Deep Dive into the C++ Implementation

Custom distance functions are not supported in pybind11-rdp because the library's core C++ implementation hard-codes Euclidean point-to-line-segment distance calculations in the LineSegment class, and the Python wrapper explicitly ignores any user-provided dist parameter with a warning.

The pybind11-rdp library provides a high-performance Python binding for the Ramer-Douglas-Peucker (RDP) polyline simplification algorithm. While the Python API accepts a dist argument for compatibility with other RDP implementations, the underlying architecture cannot accommodate custom distance metrics due to fundamental constraints in the C++ layer.

The C++ Core: Hard-Coded Distance Calculations in LineSegment

The RDP algorithm depends on calculating the distance from a point to a line segment. In pybind11-rdp, this calculation is encapsulated in the LineSegment class defined in src/main.cpp.

The class implements two fixed distance methods:

• double LineSegment::distance2(const Eigen::Vector3d &P) const – returns the squared Euclidean distance
• double LineSegment::distance(const Eigen::Vector3d &P) const – returns the standard Euclidean distance

These methods are bound to Python using pybind11 with no provision for callback injection:

.def("distance",  &LineSegment::distance,  "P"_a)
.def("distance2", &LineSegment::distance2, "P"_a)

Source: [src/main.cpp](https://github.com/cubao/pybind11-rdp/blob/master/src/main.cpp#L28-L42)

Because the distance logic is compiled into optimized C++ using Eigen vectors, there is no runtime mechanism to substitute a Python callable or even a C++ functor without modifying the source and recompiling.

The Python Wrapper: Why the dist Parameter Is Ignored

The Python layer in src/pybind11_rdp/__init__.py acknowledges the dist parameter for API compatibility but actively prevents its use. When any public function (rdp, rdp_rec, rdp_iter) receives a custom distance function, the helper __notify_dist_fn() intercepts it:

def __notify_dist_fn(dist):
    if dist is None:
        return
    print(
        "we don't support dist function, the only built‑in dist function is "
        "dist(point,line_segment) (NOT dist(point,line))",
        file=sys.stderr,
    )

Source: [src/pybind11_rdp/__init__.py](https://github.com/cubao/pybind11-rdp/blob/master/src/pybind11_rdp/__init__.py#L11-L23)

The rdp() function demonstrates this behavior explicitly:

def rdp(points, epsilon: float = 0.0, dist=None, algo="iter", return_mask=False):
    __notify_dist_fn(dist)          # prints warning if a callable is supplied

    points = np.asarray(points, dtype=np.float64)
    # ... algorithm continues with fixed C++ distance ...

    return _rdp(points, epsilon=epsilon, recursive=recursive)

Source: [src/pybind11_rdp/__init__.py](https://github.com/cubao/pybind11-rdp/blob/master/src/pybind11_rdp/__init__.py#L73-L90)

Consequently, passing a custom distance function results only in a stderr warning; the algorithm proceeds using the built-in Euclidean point-to-segment metric.

Why Adding Custom Distance Support Is Non-Trivial

Extending pybind11-rdp to support custom distance functions would require substantial architectural changes that conflict with the library's design goals.

C++ API Extension
The LineSegment class would need to accept a generic callable (e.g., std::function<double(const Eigen::Vector3d&, const LineSegment&)>) or a template functor. This would necessitate refactoring the distance methods to use dynamic dispatch rather than static inline Eigen operations.

Python-to-C++ Callback Overhead
Invoking a Python callable from within the C++ simplification loop would incur significant overhead. The RDP algorithm is recursive/iterative and calls the distance function for every candidate point; crossing the Python-C++ boundary millions of times would eliminate the performance benefits of the C++ implementation.

Algorithmic Consistency
Custom metrics might violate assumptions inherent to the RDP algorithm (e.g., triangle inequality properties). The current implementation assumes standard Euclidean geometry for point-to-segment projection calculations.

These constraints make it clear why the maintainers chose to keep the distance calculation fixed: the library prioritizes computational speed over metric flexibility.

Workarounds for Custom Metrics

If your application requires a non-Euclidean distance metric, you can approximate the behavior by post-processing the built-in simplification results.

Use the return_mask=True parameter to obtain a boolean mask of retained points, then apply your custom distance logic to filter further:

import numpy as np
from pybind11_rdp import rdp

# Step 1 – fast built‑in simplification

points = np.array([[0, 0], [1, 0.1], [2, 0.5], [3, 0.2], [4, 0]])
mask = rdp(points, epsilon=0.1, return_mask=True)

# Step 2 – apply custom metric on retained points

filtered = points[mask]

# ...custom processing here (e.g., geodesic distance checks)...

This two-stage approach preserves the performance of the C++ RDP implementation while allowing you to enforce domain-specific constraints on the simplified output.

Summary

• Hard-coded C++ implementation: The LineSegment class in src/main.cpp implements fixed Euclidean distance methods (distance() and distance2()) with no extension points for custom callables.
• Python wrapper ignores dist parameter: The __notify_dist_fn() helper in src/pybind11_rdp/__init__.py prints a warning and discards any user-provided distance function.
• Performance trade-off: Supporting custom metrics would require Python-to-C++ callback overhead that would negate the library's speed advantages.
• Workaround available: Use return_mask=True to get the built-in simplification, then apply custom distance logic in pure Python.

Frequently Asked Questions

Can I modify the C++ source to add custom distance support?

Yes, but it requires significant changes. You would need to refactor the LineSegment class in src/main.cpp to accept a std::function or template functor, modify the RDP algorithm to invoke this callable, and rebuild the package. You would also need to handle the Python binding for the callback, which introduces overhead that may degrade performance.

What distance metric does pybind11-rdp actually use?

The library uses the Euclidean distance from a point to a line segment. Specifically, it calculates the perpendicular distance (or distance to the nearest endpoint) using the LineSegment::distance() method in the C++ layer. This is not the distance from a point to an infinite line, but rather to the finite segment between two consecutive points in the polyline.

Is there a performance penalty when passing a custom dist function?

There is no runtime performance penalty because the function is never executed. The Python wrapper detects the custom callable, prints a warning to stderr via __notify_dist_fn(), and proceeds with the compiled C++ distance calculation. However, if the library were modified to actually support custom functions, significant overhead from Python-C++ boundary crossing would occur during the recursive simplification loop.

Can I use pybind11-rdp with geographic coordinates?

You can use the library with latitude/longitude arrays, but the simplification will use Euclidean distance on the raw coordinate values, not geodesic distance. For small areas where distortion is minimal, this may be acceptable. For accurate geographic simplification, you should either project coordinates to a planar CRS first, or use the return_mask=True workaround to obtain the initial simplification and then filter points using a proper geodesic distance calculation in Python.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}