# How to Use the `return_mask` Parameter in pybind11-rdp's `rdp` Function

> Learn to use the return_mask parameter in pybind11-rdp's rdp function. Get a boolean array of surviving points for easier data filtering and index tracking.

- Repository: [cubao/pybind11-rdp](https://github.com/cubao/pybind11-rdp)
- Tags: how-to-guide
- Published: 2026-02-28

---

**Set `return_mask=True` in the `rdp` function to receive a boolean array indicating which points survive the Ramer-Douglas-Peucker line simplification, enabling you to filter parallel data arrays or track original indices.**

The `rdp` function in the cubao/pybind11-rdp repository exposes a high-performance C++ implementation of the Ramer-Douglas-Peucker (RDP) algorithm through pybind11. While the default behavior returns the simplified coordinate array, the `return_mask` parameter provides a powerful alternative for workflows requiring knowledge of which specific input points were retained.

## Understanding the `return_mask` Parameter

The `return_mask` boolean argument changes the function's output type without altering the underlying simplification logic:

- **`return_mask=False` (default):** Returns an `np.ndarray` containing only the simplified coordinate points. Use this when you need the reduced polyline directly.

- **`return_mask=True`:** Returns a 1-D `np.ndarray` of dtype `bool` or `int32` (depending on the internal implementation) where `True` (or `1`) marks indices to keep. This mask aligns with the original input array's order, allowing you to filter accompanying data like timestamps, colors, or elevation values.

According to the source code in [`src/pybind11_rdp/__init__.py`](https://github.com/cubao/pybind11-rdp/blob/main/src/pybind11_rdp/__init__.py), this parameter determines whether the function calls the C++ `rdp_mask` helper or the standard `_rdp` implementation.

## Implementation Details in the Source Code

In [`src/pybind11_rdp/__init__.py`](https://github.com/cubao/pybind11-rdp/blob/main/src/pybind11_rdp/__init__.py) (lines 45–64), the Python façade handles the parameter routing:

```python
def rdp(
    points,
    epsilon: float = 0.0,
    dist=None,
    algo="iter",
    return_mask=False,
):
    """
    Main interface for the Ramer-Douglas-Peucker algorithm.
    …
    """
    __notify_dist_fn(dist)
    points = np.asarray(points, dtype=np.float64)
    recursive = "iter" != algo
    if return_mask:
        return rdp_mask(points, epsilon=epsilon, recursive=recursive)
    return _rdp(points, epsilon=epsilon, recursive=recursive)

```

When `return_mask=True`, the function invokes `rdp_mask` (exposed from the `_core` module) instead of `_rdp`. The type stub in `src/pybind11_rdp/__init__.pyi` (lines 106–119) defines this helper's signature as:

```python
def rdp_mask(
    coords: NDArray[np.float64], *, epsilon: float = 0.0, recursive: bool = True
) -> NDArray[np.int32]:
    ...

```

The `rdp_mask` function respects the `algo` parameter through the internal `recursive` flag. When `algo="iter"`, it uses the iterative implementation; when `algo="rec"`, it uses the recursive variant. Both return identical masks for the same input data.

## Practical Code Examples

### Retrieve Simplified Points (Default Behavior)

By default, `rdp` returns the filtered coordinate array. This is ideal for direct geometric processing:

```python
import numpy as np
from pybind11_rdp import rdp

pts = np.array([[0, 0], [1, 0.1], [2, 0], [3, 5], [4, 0]])
simplified = rdp(pts, epsilon=0.5)
print(simplified)

# Output: [[0, 0], [2, 0], [3, 5], [4, 0]]

```

### Obtain a Boolean Mask for Selective Indexing

Use `return_mask=True` when you need to filter parallel arrays or retain original indices:

```python
import numpy as np
from pybind11_rdp import rdp

pts = np.array([[0, 0], [1, 0.1], [2, 0], [3, 5], [4, 0]])
mask = rdp(pts, epsilon=0.5, return_mask=True)

print(mask)

# Output: [ True False  True  True False]

# Filter accompanying metadata using the same mask

colors = np.array(['red', 'green', 'blue', 'orange', 'purple'])
kept_colors = colors[mask]
print(kept_colors)

# Output: ['red' 'blue' 'orange']

```

### Combine Masks with Different Algorithms

Both iterative (`algo="iter"`) and recursive (`algo="rec"`) implementations support mask generation:

```python
import numpy as np
from pybind11_rdp import rdp

pts = np.random.rand(1000, 2)

# Generate masks using both algorithms

mask_iter = rdp(pts, epsilon=0.01, algo="iter", return_mask=True)
mask_rec = rdp(pts, epsilon=0.01, algo="rec", return_mask=True)

# Verify identical results

assert np.array_equal(mask_iter, mask_rec)

```

### Handle Extreme Epsilon Values

The mask behavior remains consistent at boundary conditions:

```python
import numpy as np
from pybind11_rdp import rdp

pts = np.cumsum(np.random.rand(20, 2), axis=0)

# Epsilon=0: no simplification, all points kept

mask_zero = rdp(pts, epsilon=0.0, return_mask=True)
print(np.all(mask_zero))  # True

# Infinite epsilon: only endpoints retained

mask_inf = rdp(pts, epsilon=np.inf, return_mask=True)
print(np.count_nonzero(mask_inf))  # 2 (first and last points only)

```

Note that the `dist` parameter is currently ignored by the implementation; passing a custom distance function triggers a warning but does not affect the mask output.

## Summary

- The `return_mask` parameter in `pybind11_rdp`'s `rdp` function switches output from simplified coordinates to a boolean index array.
- When enabled, the function dispatches to the C++ helper `rdp_mask` defined in the pybind11 bindings.
- The returned mask maintains the original input order, allowing synchronized filtering of auxiliary data arrays.
- Both iterative (`algo="iter"`) and recursive (`algo="rec"`) algorithms support mask generation with identical results.
- Edge cases like `epsilon=0.0` (full retention) and `epsilon=np.inf` (endpoint-only retention) behave predictably when using masks.

## Frequently Asked Questions

### What data type does the mask return?

The function returns a 1-D NumPy array of type `int32` (containing 0s and 1s) or `bool`, depending on the C++ implementation details. Both types work for boolean indexing in NumPy, so you can use `array[mask]` directly without casting.

### Can I use `return_mask` with custom distance functions?

Currently, no. The `dist` parameter is accepted in the function signature for API compatibility but triggers a warning and is ignored. The mask generation always uses the standard Euclidean distance metric as implemented in the C++ core.

### Does the mask preserve the original input order?

Yes. The mask array aligns index-for-index with the input `points` array. A `True` value at index `i` indicates that `points[i]` survived the simplification and should be retained. This positional alignment enables filtering of parallel time-series or attribute arrays.

### How does `return_mask` affect performance?

Generating a mask requires nearly identical computational cost to generating the simplified point array, as both operations execute the same RDP algorithm. The mask path avoids the final coordinate extraction step, making it marginally faster when you only need the boolean indices rather than the actual coordinates.