# Why Custom Distance Functions Are Not Supported in pybind11-rdp: A Deep Dive into the C++ Implementation

> Discover why pybind11-rdp disallows custom distance functions. Explore the C++ implementation details and understand the limitations of its point-to-line-segment calculations.

- Repository: [cubao/pybind11-rdp](https://github.com/cubao/pybind11-rdp)
- Tags: deep-dive
- Published: 2026-02-28

---

**Custom distance functions are not supported in pybind11-rdp because the library's core C++ implementation hard-codes Euclidean point-to-line-segment distance calculations in the `LineSegment` class, and the Python wrapper explicitly ignores any user-provided `dist` parameter with a warning.**

The `pybind11-rdp` library provides a high-performance Python binding for the Ramer-Douglas-Peucker (RDP) polyline simplification algorithm. While the Python API accepts a `dist` argument for compatibility with other RDP implementations, the underlying architecture cannot accommodate custom distance metrics due to fundamental constraints in the C++ layer.

## The C++ Core: Hard-Coded Distance Calculations in `LineSegment`

The RDP algorithm depends on calculating the distance from a point to a line segment. In `pybind11-rdp`, this calculation is encapsulated in the `LineSegment` class defined in [`src/main.cpp`](https://github.com/cubao/pybind11-rdp/blob/main/src/main.cpp).

The class implements two fixed distance methods:

- `double LineSegment::distance2(const Eigen::Vector3d &P) const` – returns the squared Euclidean distance
- `double LineSegment::distance(const Eigen::Vector3d &P) const` – returns the standard Euclidean distance

These methods are bound to Python using pybind11 with no provision for callback injection:

```cpp
.def("distance",  &LineSegment::distance,  "P"_a)
.def("distance2", &LineSegment::distance2, "P"_a)

```

*Source:* [[`src/main.cpp`](https://github.com/cubao/pybind11-rdp/blob/main/src/main.cpp)](https://github.com/cubao/pybind11-rdp/blob/master/src/main.cpp#L28-L42)

Because the distance logic is compiled into optimized C++ using Eigen vectors, there is no runtime mechanism to substitute a Python callable or even a C++ functor without modifying the source and recompiling.

## The Python Wrapper: Why the `dist` Parameter Is Ignored

The Python layer in [`src/pybind11_rdp/__init__.py`](https://github.com/cubao/pybind11-rdp/blob/main/src/pybind11_rdp/__init__.py) acknowledges the `dist` parameter for API compatibility but actively prevents its use. When any public function (`rdp`, `rdp_rec`, `rdp_iter`) receives a custom distance function, the helper `__notify_dist_fn()` intercepts it:

```python
def __notify_dist_fn(dist):
    if dist is None:
        return
    print(
        "we don't support dist function, the only built‑in dist function is "
        "dist(point,line_segment) (NOT dist(point,line))",
        file=sys.stderr,
    )

```

*Source:* [[`src/pybind11_rdp/__init__.py`](https://github.com/cubao/pybind11-rdp/blob/main/src/pybind11_rdp/__init__.py)](https://github.com/cubao/pybind11-rdp/blob/master/src/pybind11_rdp/__init__.py#L11-L23)

The `rdp()` function demonstrates this behavior explicitly:

```python
def rdp(points, epsilon: float = 0.0, dist=None, algo="iter", return_mask=False):
    __notify_dist_fn(dist)          # prints warning if a callable is supplied

    points = np.asarray(points, dtype=np.float64)
    # ... algorithm continues with fixed C++ distance ...

    return _rdp(points, epsilon=epsilon, recursive=recursive)

```

*Source:* [[`src/pybind11_rdp/__init__.py`](https://github.com/cubao/pybind11-rdp/blob/main/src/pybind11_rdp/__init__.py)](https://github.com/cubao/pybind11-rdp/blob/master/src/pybind11_rdp/__init__.py#L73-L90)

Consequently, passing a custom distance function results only in a stderr warning; the algorithm proceeds using the built-in Euclidean point-to-segment metric.

## Why Adding Custom Distance Support Is Non-Trivial

Extending `pybind11-rdp` to support custom distance functions would require substantial architectural changes that conflict with the library's design goals.

**C++ API Extension**  
The `LineSegment` class would need to accept a generic callable (e.g., `std::function<double(const Eigen::Vector3d&, const LineSegment&)>`) or a template functor. This would necessitate refactoring the distance methods to use dynamic dispatch rather than static inline Eigen operations.

**Python-to-C++ Callback Overhead**  
Invoking a Python callable from within the C++ simplification loop would incur significant overhead. The RDP algorithm is recursive/iterative and calls the distance function for every candidate point; crossing the Python-C++ boundary millions of times would eliminate the performance benefits of the C++ implementation.

**Algorithmic Consistency**  
Custom metrics might violate assumptions inherent to the RDP algorithm (e.g., triangle inequality properties). The current implementation assumes standard Euclidean geometry for point-to-segment projection calculations.

These constraints make it clear why the maintainers chose to keep the distance calculation fixed: **the library prioritizes computational speed over metric flexibility**.

## Workarounds for Custom Metrics

If your application requires a non-Euclidean distance metric, you can approximate the behavior by post-processing the built-in simplification results.

Use the `return_mask=True` parameter to obtain a boolean mask of retained points, then apply your custom distance logic to filter further:

```python
import numpy as np
from pybind11_rdp import rdp

# Step 1 – fast built‑in simplification

points = np.array([[0, 0], [1, 0.1], [2, 0.5], [3, 0.2], [4, 0]])
mask = rdp(points, epsilon=0.1, return_mask=True)

# Step 2 – apply custom metric on retained points

filtered = points[mask]

# ...custom processing here (e.g., geodesic distance checks)...

```

This two-stage approach preserves the performance of the C++ RDP implementation while allowing you to enforce domain-specific constraints on the simplified output.

## Summary

- **Hard-coded C++ implementation**: The `LineSegment` class in [`src/main.cpp`](https://github.com/cubao/pybind11-rdp/blob/main/src/main.cpp) implements fixed Euclidean distance methods (`distance()` and `distance2()`) with no extension points for custom callables.
- **Python wrapper ignores `dist` parameter**: The `__notify_dist_fn()` helper in [`src/pybind11_rdp/__init__.py`](https://github.com/cubao/pybind11-rdp/blob/main/src/pybind11_rdp/__init__.py) prints a warning and discards any user-provided distance function.
- **Performance trade-off**: Supporting custom metrics would require Python-to-C++ callback overhead that would negate the library's speed advantages.
- **Workaround available**: Use `return_mask=True` to get the built-in simplification, then apply custom distance logic in pure Python.

## Frequently Asked Questions

### Can I modify the C++ source to add custom distance support?

Yes, but it requires significant changes. You would need to refactor the `LineSegment` class in [`src/main.cpp`](https://github.com/cubao/pybind11-rdp/blob/main/src/main.cpp) to accept a `std::function` or template functor, modify the RDP algorithm to invoke this callable, and rebuild the package. You would also need to handle the Python binding for the callback, which introduces overhead that may degrade performance.

### What distance metric does pybind11-rdp actually use?

The library uses the **Euclidean distance from a point to a line segment**. Specifically, it calculates the perpendicular distance (or distance to the nearest endpoint) using the `LineSegment::distance()` method in the C++ layer. This is **not** the distance from a point to an infinite line, but rather to the finite segment between two consecutive points in the polyline.

### Is there a performance penalty when passing a custom dist function?

There is no runtime performance penalty because the function is never executed. The Python wrapper detects the custom callable, prints a warning to stderr via `__notify_dist_fn()`, and proceeds with the compiled C++ distance calculation. However, if the library were modified to actually support custom functions, significant overhead from Python-C++ boundary crossing would occur during the recursive simplification loop.

### Can I use pybind11-rdp with geographic coordinates?

You can use the library with latitude/longitude arrays, but the simplification will use Euclidean distance on the raw coordinate values, not geodesic distance. For small areas where distortion is minimal, this may be acceptable. For accurate geographic simplification, you should either project coordinates to a planar CRS first, or use the `return_mask=True` workaround to obtain the initial simplification and then filter points using a proper geodesic distance calculation in Python.