How to Use meshopt_generateVertexRemapCustom for Floating-Point Tolerance When Merging Vertices

Use meshopt_generateVertexRemapCustom with a custom equality callback that compares vertex positions within an epsilon distance, allowing the function to merge vertices that differ only by floating-point noise while using exact hashing for performance.

The meshoptimizer library by zeux provides low-level tools for mesh optimization, including vertex deduplication. When importing meshes from external tools, floating-point precision issues often create duplicate vertices that are not exactly bitwise equal. The meshopt_generateVertexRemapCustom function declared in src/meshoptimizer.h solves this by allowing a custom equality callback that can merge vertices within a floating-point tolerance.

Understanding the Custom Remap API

Unlike the standard vertex remap functions, meshopt_generateVertexRemapCustom exposes a two-stage hashing process implemented in src/indexgenerator.cpp. The function first uses VertexCustomHasher::hash to group vertices by the integer representation of their three 32-bit position components. This creates fast initial buckets of potentially duplicate vertices.

After a hash collision occurs, the code falls back to VertexCustomHasher::equal. This method first checks for exact bitwise equality of the position components. If that fails and a callback is provided, it invokes your custom function to determine if the vertices should be merged despite small floating-point differences.

The function signature from src/meshoptimizer.h:

size_t meshopt_generateVertexRemapCustom(
    unsigned int* destination,
    const unsigned int* indices,
    size_t index_count,
    const float* vertex_positions,
    size_t vertex_count,
    size_t vertex_positions_stride,
    int (*callback)(void*, unsigned int, unsigned int),
    void* context);

The callback receives a user-defined context pointer and two vertex indices, returning non-zero if the vertices are acceptable matches.

Implementing a Tolerance-Based Equality Check

To merge vertices within a floating-point epsilon, you must provide a context structure and a callback function that calculates the distance between candidate vertices.

Define a Context Structure

Create a structure to hold your tolerance threshold and position data access patterns:

struct ToleranceContext {
    float epsilon;           // Distance threshold, e.g., 1e-5f
    const float* positions;  // Pointer to vertex position data
    size_t stride_floats;    // Stride in floats (vertex_positions_stride / sizeof(float))
};

Write the Equality Callback

Implement the callback to compute the squared Euclidean distance between two vertex positions. Return non-zero if they are within the tolerance:

int position_tolerance_callback(void* ctx, unsigned int a, unsigned int b) {
    const ToleranceContext* t = (const ToleranceContext*)ctx;
    const float* pa = t->positions + a * t->stride_floats;
    const float* pb = t->positions + b * t->stride_floats;
    
    float dx = pa[0] - pb[0];
    float dy = pa[1] - pb[1];
    float dz = pa[2] - pb[2];
    float dist_sq = dx*dx + dy*dy + dz*dz;
    
    return dist_sq <= (t->epsilon * t->epsilon); // Non-zero means "equal"
}

Invoke the API

Call meshopt_generateVertexRemapCustom with your callback and context:

// Prepare context
ToleranceContext ctx;
ctx.epsilon = 1e-5f;
ctx.positions = vertex_positions;
ctx.stride_floats = vertex_positions_stride / sizeof(float);

// Allocate remap table (one entry per original vertex)
std::vector<unsigned int> remap(vertex_count);

// Generate remap
size_t unique_count = meshopt_generateVertexRemapCustom(
    remap.data(),
    indices,           // Can be nullptr if not using an index buffer
    index_count,
    vertex_positions,
    vertex_count,
    vertex_positions_stride,
    position_tolerance_callback,
    &ctx);

Applying the Vertex Remap

Once the remap table is generated, use meshopt_remapVertexBuffer and meshopt_remapIndexBuffer to create compacted buffers without duplicate vertices:

// Compact vertices
std::vector<Vertex> compacted_vertices(unique_count);
meshopt_remapVertexBuffer(
    compacted_vertices.data(),
    vertices,
    vertex_count,
    sizeof(Vertex),
    remap.data());

// Remap indices to point to new vertex locations
std::vector<unsigned int> remapped_indices(index_count);
meshopt_remapIndexBuffer(
    remapped_indices.data(),
    indices,
    index_count,
    remap.data());

Performance Characteristics

The implementation in src/indexgenerator.cpp optimizes performance by only invoking your callback for vertices that hash to the same bucket. Since the hash is based on the raw integer bits of the position components, the tolerance test is only performed on candidates that are already close in memory representation. This makes the floating-point tolerance check cheap even on large meshes, as the expensive distance calculation is guarded by fast hash comparisons.

Summary

• meshopt_generateVertexRemapCustom allows custom equality logic through a callback mechanism defined in src/meshoptimizer.h.
• The function uses two-stage hashing: first by integer representation of positions in VertexCustomHasher::hash, then by your custom tolerance callback in VertexCustomHasher::equal.
• Implement a context structure to store your epsilon value and a callback that returns non-zero for vertices within the desired distance.
• Apply the resulting remap table with meshopt_remapVertexBuffer and meshopt_remapIndexBuffer to produce optimized mesh data.

Frequently Asked Questions

How does the hashing work with floating-point tolerance?

The function hashes vertices based on the exact integer representation of their three 32-bit float components in VertexCustomHasher::hash. Only vertices that collide in this hash bucket proceed to the equality test. If the exact bitwise comparison fails, your custom callback is invoked to determine if the floating-point difference is within acceptable tolerance. This ensures the expensive distance calculation only runs on geometrically close candidates.

Can I use this function without an index buffer?

Yes. Pass nullptr for the indices parameter and 0 for index_count. The function will generate a remap table for all vertices in the vertex_positions array based solely on position proximity within your tolerance.

What should the callback return for vertices that should be merged?

The callback should return non-zero (true) if the two vertices should be considered identical and merged together. Return zero (false) if they represent distinct vertices that happen to have hash collisions but are outside your tolerance distance.

Is there a performance penalty for using a custom callback?

The callback is only invoked when hash collisions occur and exact equality checks fail. Since the hash groups vertices by their integer bit patterns, the callback is rarely called for vertices that are far apart in space. According to the implementation in src/indexgenerator.cpp, this makes the tolerance-based merge nearly as fast as exact matching while handling floating-point precision issues.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}