# Memory Efficiency When Handling Large Point Clouds with pybind11-rdp: Zero-Copy Strategies Explained > Boost memory efficiency with pybind11-rdp for large point clouds. Discover zero-copy strategies like NumPy views and boolean masks to avoid intermediate copies and optimize processing. - Repository: [cubao/pybind11-rdp](https://github.com/cubao/pybind11-rdp) - Tags: performance - Published: 2026-02-28 --- **pybind11-rdp minimizes memory overhead when processing million-point clouds by leveraging zero-copy NumPy views via Eigen references, iterative stack-based traversal, and boolean mask workflows that avoid intermediate coordinate copies.** When simplifying massive 2-D or 3-D point clouds using the Ramer-Douglas-Peucker (RDP) algorithm, memory constraints often become the bottleneck before CPU speed. The `cubao/pybind11-rdp` library addresses this through careful C++ design choices exposed to Python via pybind11, ensuring that **memory efficiency handling large point clouds with pybind11-rdp** remains optimal even beyond ten million points. ## Zero-Copy Data Bridging with Eigen::Ref The library eliminates unnecessary memory allocations at the Python-C++ boundary by accepting `Eigen::Ref` arguments in its public bindings. This lightweight view wraps existing NumPy buffers without copying the underlying data. In [`src/main.cpp`](https://github.com/cubao/pybind11-rdp/blob/main/src/main.cpp), the core `rdp` binding receives input arrays through this mechanism: ```cpp // src/main.cpp lines 15-23 RowVectors rdp(Eigen::Ref points, double epsilon, ...); ``` Because `Eigen::Ref` acts as a view rather than an owner, the C++ core operates directly on the Python array's memory. This zero-copy bridge ensures that passing a 5-million-point array from Python consumes no additional RAM beyond the original buffer. ## Iterative Traversal for Constant Stack Memory Recursive RDP implementations risk stack overflow with long polylines due to unbounded call depth. The library provides `douglas_simplify_iter` as an alternative to the recursive `douglas_simplify`, using an explicit `std::queue` to manage index pairs: ```cpp // src/main.cpp lines 86-126 while (!queue.empty()) { auto [start, end] = queue.front(); queue.pop(); // ... process segment ... } ``` This iterative approach bounds temporary memory to **O(1)** per iteration rather than consuming stack frames for each recursion level. For balanced splits, the queue holds at most **O(log N)** index pairs, making it safe for arbitrarily deep curves without risking segmentation faults. ## Mask-Based Selection Workflows Rather than constructing new point arrays during simplification, the algorithm first generates a boolean mask through `douglas_simplify_mask`, then converts it to indices via `mask2indexes`, and finally compacts the result with `select_by_mask`. This three-phase pipeline avoids intermediate coordinate copies: ```cpp // src/main.cpp lines 28-71 Eigen::VectorXi douglas_simplify_mask(...); std::vector mask2indexes(const Eigen::VectorXi &mask); RowVectors select_by_mask(Eigen::Ref points, ...); ``` Users can halt the pipeline early by requesting only the mask via `rdp_mask`, which returns a compact `Eigen::VectorXi` of `int32` values rather than a full coordinate array. This is implemented in [`src/main.cpp`](https://github.com/cubao/pybind11-rdp/blob/main/src/main.cpp) lines 35-58 and exposed through the Python API with the `return_mask` parameter. ## Row-Major Storage Alignment The `RowVectors` type alias in [`src/main.cpp`](https://github.com/cubao/pybind11-rdp/blob/main/src/main.cpp) lines 46-48 defines matrices as row-major: ```cpp typedef Eigen::Matrix RowVectors; ``` This layout matches NumPy's default C-contiguous ordering, allowing the Python side to reuse the underlying buffer directly without transposition costs or layout conversions. ## Python API Design for Memory Constraints The high-level interface in [`src/pybind11_rdp/__init__.py`](https://github.com/cubao/pybind11-rdp/blob/main/src/pybind11_rdp/__init__.py) (lines 25-95) forwards NumPy data to the C++ core without additional copies. The `return_mask` flag enables zero-copy workflows where users apply the boolean mask to their original array as a view rather than generating a new coordinate set: ```python from pybind11_rdp import rdp_mask mask = rdp_mask(points, epsilon=0.02) # Returns compact int32 mask kept_points = points[mask.astype(bool)] # View, not copy ``` ## Working with Massive Point Clouds: Code Examples ### Processing 5 Million Points Without Duplication ```python import numpy as np from pybind11_rdp import rdp # Simulate a large point cloud: 5 million 3-D points pts = np.random.rand(5_000_000, 3).astype(np.float64) # Zero-copy processing with iterative algorithm simplified = rdp(pts, epsilon=0.01, algo="iter") print(f"Original: {pts.shape}, Simplified: {simplified.shape}") ``` ### Mask-Only Workflow for Memory Sensitive Applications ```python import numpy as np from pybind11_rdp import rdp_mask # Load massive cloud from file pts = np.loadtxt("large_cloud.xyz") # shape (N, 3) # Generate mask without allocating new coordinate array mask = rdp_mask(pts, epsilon=0.02) kept = pts[mask.astype(bool)] # Optional view of kept points print(f"Kept points: {kept.shape[0]} out of {pts.shape[0]}") ``` ### Iterative vs. Recursive Safety Comparison ```python from pybind11_rdp import rdp import numpy as np # Synthetic polyline with 1 million collinear points line = np.column_stack([np.arange(1_000_000), np.zeros(1_000_000)]) # Recursive: risks stack overflow on extreme inputs simpl_rec = rdp(line, epsilon=0.0, algo="rec") # Iterative: constant memory regardless of input depth simpl_iter = rdp(line, epsilon=0.0, algo="iter") ``` ## Summary - **Zero-copy bridging**: `Eigen::Ref` in [`src/main.cpp`](https://github.com/cubao/pybind11-rdp/blob/main/src/main.cpp) eliminates data duplication between Python and C++. - **Flat stack processing**: The iterative `douglas_simplify_iter` uses `std::queue` to limit memory to **O(log N)** entries instead of recursive stack growth. - **Mask-based pipelines**: `rdp_mask` and `mask2indexes` allow workflows that avoid intermediate coordinate copies, operating via boolean selections and views. - **Storage alignment**: Row-major `RowVectors` type ensures NumPy buffer compatibility without layout conversion overhead. - **Selective API**: The `return_mask` parameter in [`src/pybind11_rdp/__init__.py`](https://github.com/cubao/pybind11-rdp/blob/main/src/pybind11_rdp/__init__.py) enables users to maintain single-source-of-truth for point data. ## Frequently Asked Questions ### Does pybind11-rdp copy my NumPy array into C++ memory? No. The library uses `Eigen::Ref` to create a lightweight view of your existing NumPy buffer. As implemented in [`src/main.cpp`](https://github.com/cubao/pybind11-rdp/blob/main/src/main.cpp) lines 15-23, the C++ functions read directly from the Python-managed memory without allocation or duplication, ensuring zero-copy overhead when passing arrays from Python. ### What is the memory complexity of the iterative RDP algorithm? The iterative implementation uses **O(log N)** auxiliary memory in the worst case, stored in an explicit `std::queue` of index pairs. Unlike the recursive version `douglas_simplify`, which consumes stack frames proportional to recursion depth, the iterative `douglas_simplify_iter` (lines 86-126) maintains constant memory per iteration, preventing stack overflow on million-point inputs. ### Can I process point clouds larger than my available RAM? No. While pybind11-rdp is memory-efficient, the entire point cloud must fit in RAM as a contiguous NumPy array, since the library operates on in-memory Eigen references. The efficiency gains come from avoiding temporary copies during processing, not from streaming or out-of-core algorithms. For datasets exceeding RAM, consider tiled processing or memory-mapped arrays that load subsets into pybind11-rdp sequentially. ### When should I use `rdp_mask` instead of the standard `rdp` function? Use `rdp_mask` when you need to preserve the original array or apply the simplification logic to multiple related arrays (such as color or intensity channels). The mask is a compact `int32` vector representing kept indices, allowing you to index into your original data as a view rather than allocating a new coordinate array, as shown in the mask-based workflow example.