# Understanding the Eigen VectorXi Mask Output Format in pybind11-rdp

> Learn how pybind11-rdp uses Eigen VectorXi for rdp_mask output. Understand binary masks and retained points in polyline simplification.

- Repository: [cubao/pybind11-rdp](https://github.com/cubao/pybind11-rdp)
- Tags: deep-dive
- Published: 2026-02-28

---

**The `rdp_mask` function in pybind11-rdp returns an `Eigen::VectorXi` binary mask where each element is 1 if the corresponding point is retained in the simplified polyline or 0 if it is discarded.**

The `cubao/pybind11-rdp` library provides high-performance Ramer-Douglas-Peucker line simplification through pybind11. When you need to identify which specific input points survive the simplification process—rather than receiving only the simplified coordinates—the library exposes its internal representation as an **Eigen VectorXi mask output format**.

## What is the Eigen VectorXi Mask Output Format?

The mask is a dense integer vector aligned one-to-one with your input coordinate array. According to the implementation in [`src/main.cpp`](https://github.com/cubao/pybind11-rdp/blob/main/src/main.cpp), this format serves as a binary indicator where vector positions mark retention status during the Douglas-Peucker algorithm.

### Memory Allocation and Zero-Initialization

The mask allocation occurs at the entry point of the simplification routine. The code constructs a vector matching the row dimension of the input coordinate matrix and initializes all entries to zero:

```cpp
Eigen::VectorXi mask(coords.rows());
mask.setZero();                              // ← zero‑initialise

```

(source: [src/main.cpp lines 132-134](https://github.com/cubao/pybind11-rdp/blob/master/src/main.cpp#L132-L134))

### Population via Recursive or Iterative RDP

The library fills the mask by invoking either the recursive or iterative variant of the algorithm, passing the mask by reference for in-place modification:

```cpp
if (recursive) {
    douglas_simplify(coords, mask, 0, mask.size() - 1, epsilon);
} else {
    douglas_simplify_iter(coords, mask, epsilon);
}

```

(source: [src/main.cpp lines 134-138](https://github.com/cubao/pybind11-rdp/blob/master/src/main.cpp#L134-L138))

During execution, both algorithms mark retained segment endpoints by setting specific indices to 1:

```cpp
to_keep[i] = to_keep[j] = 1;                // ← set mask entries to 1

```

(source: [src/main.cpp lines 51-55](https://github.com/cubao/pybind11-rdp/blob/master/src/main.cpp#L51-L55))

## Converting Masks to Index Arrays

While the binary mask efficiently represents retention state, applications often require the actual indices of kept points. The library implements `mask2indexes` in [`src/main.cpp`](https://github.com/cubao/pybind11-rdp/blob/main/src/main.cpp) to perform this conversion by iterating through the mask and collecting positions marked with 1:

```cpp
Eigen::VectorXi indexes(mask.sum());
for (int i = 0, j = 0, N = mask.size(); i < N; ++i) {
    if (mask[i]) {
        indexes[j++] = i;
    }
}

```

(source: [src/main.cpp lines 142-149](https://github.com/cubao/pybind11-rdp/blob/master/src/main.cpp#L142-L149))

The high-level `rdp` function internally utilizes `mask2indexes` to return the simplified coordinate array, whereas `rdp_mask` exposes the raw binary mask directly to Python.

## Working with Mask Output in Python

The following examples demonstrate practical workflows using the **Eigen VectorXi mask output format** returned by `pybind11_rdp.rdp_mask`.

### Basic Mask Inspection

```python
import numpy as np
from pybind11_rdp import rdp_mask

coords = np.array([[i, np.sin(i)] for i in np.linspace(0, 2*np.pi, 10)])
mask = rdp_mask(coords)

print("Mask:", mask)
print("Kept points:", coords[mask.astype(bool)])

```

### Extracting Indices with NumPy

```python
import numpy as np
from pybind11_rdp import rdp_mask

coords = np.random.rand(20, 2)
mask = rdp_mask(coords, epsilon=0.1)
indices = np.flatnonzero(mask)

print("Kept indices:", indices)

```

### Synchronizing Mask with Auxiliary Data

```python
import numpy as np
from pybind11_rdp import rdp_mask

coords = np.random.rand(50, 2)
timestamps = np.arange(50)

mask = rdp_mask(coords, epsilon=0.05)
filtered_coords = coords[mask.astype(bool)]
filtered_timestamps = timestamps[mask.astype(bool)]

```

## Summary

- The **Eigen VectorXi mask** is a binary integer vector where 1 indicates retention and 0 indicates removal during polyline simplification.
- The mask is allocated in [`src/main.cpp`](https://github.com/cubao/pybind11-rdp/blob/main/src/main.cpp) with `setZero()` initialization at lines 132-134.
- Both `douglas_simplify` and `douglas_simplify_iter` populate the mask by setting segment endpoints to 1 during the RDP recursion (lines 51-55 and 134-138).
- Use `mask2indexes` (or NumPy's `flatnonzero`) to convert the binary mask to an array of retained indices.
- The `rdp_mask` Python binding returns this format directly, enabling flexible filtering of associated datasets synchronized with your geometry.

## Frequently Asked Questions

### What data type does rdp_mask return?

The function returns a NumPy array of dtype `int32` (corresponding to C++ `int`), which is the Python representation of `Eigen::VectorXi`. Each element is guaranteed to be either 0 or 1.

### How do I get the indices of kept points from the mask?

Use `numpy.flatnonzero(mask)` to obtain an array of indices where the mask equals 1. This mirrors the C++ `mask2indexes` function implemented in [`src/main.cpp`](https://github.com/cubao/pybind11-rdp/blob/main/src/main.cpp) lines 142-149.

### Why does the mask use 1 and 0 instead of boolean values?

The `Eigen::VectorXi` type stores 32-bit integers for compatibility with Eigen's dense vector operations and to facilitate efficient summation operations (such as `mask.sum()` used during index conversion) without type casting overhead.

### Can I use the mask to filter other arrays synchronized with my coordinates?

Yes. Since the mask length matches the input point count exactly, you can apply boolean indexing to any parallel array—such as timestamps, sensor readings, or elevations—using `mask.astype(bool)` to maintain data synchronization with the simplified geometry.