Memory Efficiency When Handling Large Point Clouds with pybind11-rdp: Zero-Copy Strategies Explained

pybind11-rdp minimizes memory overhead when processing million-point clouds by leveraging zero-copy NumPy views via Eigen references, iterative stack-based traversal, and boolean mask workflows that avoid intermediate coordinate copies.

When simplifying massive 2-D or 3-D point clouds using the Ramer-Douglas-Peucker (RDP) algorithm, memory constraints often become the bottleneck before CPU speed. The cubao/pybind11-rdp library addresses this through careful C++ design choices exposed to Python via pybind11, ensuring that memory efficiency handling large point clouds with pybind11-rdp remains optimal even beyond ten million points.

Zero-Copy Data Bridging with Eigen::Ref

The library eliminates unnecessary memory allocations at the Python-C++ boundary by accepting Eigen::Ref<const RowVectors> arguments in its public bindings. This lightweight view wraps existing NumPy buffers without copying the underlying data.

In src/main.cpp, the core rdp binding receives input arrays through this mechanism:

// src/main.cpp lines 15-23
RowVectors rdp(Eigen::Ref<const RowVectors> points, double epsilon, ...);

Because Eigen::Ref acts as a view rather than an owner, the C++ core operates directly on the Python array's memory. This zero-copy bridge ensures that passing a 5-million-point array from Python consumes no additional RAM beyond the original buffer.

Iterative Traversal for Constant Stack Memory

Recursive RDP implementations risk stack overflow with long polylines due to unbounded call depth. The library provides douglas_simplify_iter as an alternative to the recursive douglas_simplify, using an explicit std::queue to manage index pairs:

// src/main.cpp lines 86-126
while (!queue.empty()) {
    auto [start, end] = queue.front();
    queue.pop();
    // ... process segment ...
}

This iterative approach bounds temporary memory to O(1) per iteration rather than consuming stack frames for each recursion level. For balanced splits, the queue holds at most O(log N) index pairs, making it safe for arbitrarily deep curves without risking segmentation faults.

Mask-Based Selection Workflows

Rather than constructing new point arrays during simplification, the algorithm first generates a boolean mask through douglas_simplify_mask, then converts it to indices via mask2indexes, and finally compacts the result with select_by_mask. This three-phase pipeline avoids intermediate coordinate copies:

// src/main.cpp lines 28-71
Eigen::VectorXi douglas_simplify_mask(...);
std::vector<int> mask2indexes(const Eigen::VectorXi &mask);
RowVectors select_by_mask(Eigen::Ref<const RowVectors> points, ...);

Users can halt the pipeline early by requesting only the mask via rdp_mask, which returns a compact Eigen::VectorXi of int32 values rather than a full coordinate array. This is implemented in src/main.cpp lines 35-58 and exposed through the Python API with the return_mask parameter.

Row-Major Storage Alignment

The RowVectors type alias in src/main.cpp lines 46-48 defines matrices as row-major:

typedef Eigen::Matrix<double, Eigen::Dynamic, 3, Eigen::RowMajor> RowVectors;

This layout matches NumPy's default C-contiguous ordering, allowing the Python side to reuse the underlying buffer directly without transposition costs or layout conversions.

Python API Design for Memory Constraints

The high-level interface in src/pybind11_rdp/__init__.py (lines 25-95) forwards NumPy data to the C++ core without additional copies. The return_mask flag enables zero-copy workflows where users apply the boolean mask to their original array as a view rather than generating a new coordinate set:

from pybind11_rdp import rdp_mask

mask = rdp_mask(points, epsilon=0.02)  # Returns compact int32 mask

kept_points = points[mask.astype(bool)]  # View, not copy

Working with Massive Point Clouds: Code Examples

Processing 5 Million Points Without Duplication

import numpy as np
from pybind11_rdp import rdp

# Simulate a large point cloud: 5 million 3-D points

pts = np.random.rand(5_000_000, 3).astype(np.float64)

# Zero-copy processing with iterative algorithm

simplified = rdp(pts, epsilon=0.01, algo="iter")
print(f"Original: {pts.shape}, Simplified: {simplified.shape}")

Mask-Only Workflow for Memory Sensitive Applications

import numpy as np
from pybind11_rdp import rdp_mask

# Load massive cloud from file

pts = np.loadtxt("large_cloud.xyz")  # shape (N, 3)

# Generate mask without allocating new coordinate array

mask = rdp_mask(pts, epsilon=0.02)
kept = pts[mask.astype(bool)]  # Optional view of kept points

print(f"Kept points: {kept.shape[0]} out of {pts.shape[0]}")

Iterative vs. Recursive Safety Comparison

from pybind11_rdp import rdp
import numpy as np

# Synthetic polyline with 1 million collinear points

line = np.column_stack([np.arange(1_000_000), np.zeros(1_000_000)])

# Recursive: risks stack overflow on extreme inputs

simpl_rec = rdp(line, epsilon=0.0, algo="rec")

# Iterative: constant memory regardless of input depth

simpl_iter = rdp(line, epsilon=0.0, algo="iter")

Summary

• Zero-copy bridging: Eigen::Ref<const RowVectors> in src/main.cpp eliminates data duplication between Python and C++.
• Flat stack processing: The iterative douglas_simplify_iter uses std::queue to limit memory to O(log N) entries instead of recursive stack growth.
• Mask-based pipelines: rdp_mask and mask2indexes allow workflows that avoid intermediate coordinate copies, operating via boolean selections and views.
• Storage alignment: Row-major RowVectors type ensures NumPy buffer compatibility without layout conversion overhead.
• Selective API: The return_mask parameter in src/pybind11_rdp/__init__.py enables users to maintain single-source-of-truth for point data.

Frequently Asked Questions

Does pybind11-rdp copy my NumPy array into C++ memory?

No. The library uses Eigen::Ref<const RowVectors> to create a lightweight view of your existing NumPy buffer. As implemented in src/main.cpp lines 15-23, the C++ functions read directly from the Python-managed memory without allocation or duplication, ensuring zero-copy overhead when passing arrays from Python.

What is the memory complexity of the iterative RDP algorithm?

The iterative implementation uses O(log N) auxiliary memory in the worst case, stored in an explicit std::queue of index pairs. Unlike the recursive version douglas_simplify, which consumes stack frames proportional to recursion depth, the iterative douglas_simplify_iter (lines 86-126) maintains constant memory per iteration, preventing stack overflow on million-point inputs.

Can I process point clouds larger than my available RAM?

No. While pybind11-rdp is memory-efficient, the entire point cloud must fit in RAM as a contiguous NumPy array, since the library operates on in-memory Eigen references. The efficiency gains come from avoiding temporary copies during processing, not from streaming or out-of-core algorithms. For datasets exceeding RAM, consider tiled processing or memory-mapped arrays that load subsets into pybind11-rdp sequentially.

When should I use rdp_mask instead of the standard rdp function?

Use rdp_mask when you need to preserve the original array or apply the simplification logic to multiple related arrays (such as color or intensity channels). The mask is a compact int32 vector representing kept indices, allowing you to index into your original data as a view rather than allocating a new coordinate array, as shown in the mask-based workflow example.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}