How to Handle Degenerate and Duplicate Triangles with meshopt_filterIndexBuffer

TL;DR: meshopt_filterIndexBuffer removes degenerate triangles (where two or more vertices share identical position data) and duplicate triangles (same three vertex keys and winding order) from an index buffer, returning a new index count while preserving opposite-winding duplicates for double-sided rendering.

The meshoptimizer library by Zeux provides optimized geometry processing utilities for real-time rendering. When preparing mesh data for the GPU, index buffers often contain triangles that contribute nothing to the visible surface or waste GPU cycles due to redundancy. The meshopt_filterIndexBuffer function, declared in src/meshoptimizer.h and implemented in src/indexgenerator.cpp, efficiently cleans these artifacts by comparing only the first vertex_size bytes of each vertex—typically the position attribute—to determine uniqueness.

Understanding Degenerate and Duplicate Triangles

Degenerate Triangles

A degenerate triangle occurs when two or more of its vertices share the same vertex key, meaning they refer to identical vertex data. These triangles have zero area and produce no visible output, yet they still consume GPU resources during rasterization. According to the implementation in src/indexgenerator.cpp, the filter identifies these by comparing the hashed vertex keys of each triangle's three indices.

Duplicate Triangles and Winding Order

Duplicate triangles are defined as those that have already appeared earlier in the index stream with the same three vertex keys and the same winding order. The function preserves duplicate triangles that have opposite winding because they can be required for double-sided rendering or mesh consistency. This behavior is hardcoded in the filtering logic to ensure that flipping a triangle's normal does not cause it to be culled during the cleaning process.

The meshopt_filterIndexBuffer API

The basic function signature accepts a destination buffer, source indices, vertex data, and the size of the vertex key to use for comparison:

size_t meshopt_filterIndexBuffer(
    unsigned int* destination,
    const unsigned int* indices,
    size_t index_count,
    const void* vertices,
    size_t vertex_count,
    size_t vertex_size,
    size_t vertex_stride
);

The vertex_size parameter specifies how many bytes of each vertex to compare—typically sizeof(float) * 3 for position-only meshes—while vertex_stride is the byte distance between consecutive vertices in the buffer. The function returns the new index count after filtering, which is always less than or equal to the original index_count.

Handling Complex Meshes with meshopt_filterIndexBufferMulti

For meshes where uniqueness depends on more than just position—such as skinned meshes with bone indices and weights—the meshopt_filterIndexBufferMulti variant accepts multiple vertex streams. This allows you to define exactly which attributes constitute a unique vertex.

struct meshopt_Stream {
    const void* data;
    size_t size;
    size_t stride;
};

size_t meshopt_filterIndexBufferMulti(
    unsigned int* destination,
    const unsigned int* indices,
    size_t index_count,
    size_t vertex_count,
    const struct meshopt_Stream* streams,
    size_t stream_count
);

Each meshopt_Stream specifies a data pointer, the number of bytes to read from that stream (size), and the stride between elements. The function concatenates these byte ranges to form the complete vertex key used for comparison.

Implementation Details from Source Code

In src/indexgenerator.cpp, the filtering algorithm builds a hash table of vertex keys to detect duplicates in linear time. The implementation processes each triangle sequentially, checking if all three vertices are unique within that triangle (to catch degenerates) and if the triangle's key combination has been seen before (to catch duplicates). The source code explicitly skips triangles with zero area by comparing indices after vertex key lookup, ensuring that positionally identical vertices are treated as a single point regardless of their index values.

Practical Code Examples

Basic Position-Based Filtering

For a standard mesh where only position determines triangle uniqueness, allocate a destination buffer at least as large as the source and filter based on the first 12 bytes (three floats) of each vertex:

#include <meshoptimizer.h>
#include <vector>

// Assume Vertex starts with a float3 position
struct Vertex {
    float position[3];
    float normal[3];
    float uv[2];
};

std::vector<unsigned int> filterMesh(
    const std::vector<unsigned int>& indices,
    const std::vector<Vertex>& vertices)
{
    std::vector<unsigned int> filtered(indices.size());
    
    size_t newCount = meshopt_filterIndexBuffer(
        filtered.data(),
        indices.data(),
        indices.size(),
        vertices.data(),
        vertices.size(),
        sizeof(float) * 3,  // Compare only position
        sizeof(Vertex));    // Full vertex stride
    
    filtered.resize(newCount);
    return filtered;
}

Filtering Skinned Meshes with Multiple Streams

When vertices share the same position but differ in bone weights, use the multi-stream API to include both position and skinning data in the equality check:

std::vector<unsigned int> filterSkinnedMesh(
    const std::vector<unsigned int>& indices,
    const std::vector<Vertex>& vertices,
    const void* boneIndices,  // e.g., uint8_t[4] per vertex
    const void* boneWeights)  // e.g., uint8_t[4] per vertex
{
    std::vector<unsigned int> filtered(indices.size());
    
    meshopt_Stream streams[2];
    streams[0].data = vertices.data();
    streams[0].size = sizeof(float) * 3;  // Position only
    streams[0].stride = sizeof(Vertex);
    
    streams[1].data = boneIndices;
    streams[1].size = 4;  // Four bone indices
    streams[1].stride = sizeof(Vertex);  // Assuming interleaved or same stride
    
    size_t newCount = meshopt_filterIndexBufferMulti(
        filtered.data(),
        indices.data(),
        indices.size(),
        vertices.size(),
        streams,
        2);
    
    filtered.resize(newCount);
    return filtered;
}

Integration with Vertex Remapping

Combine filtering with meshopt_generateVertexRemap to produce a compact, render-ready mesh using the pipeline demonstrated in demo/main.cpp:

// 1. Generate vertex remap to remove unused vertices
std::vector<unsigned int> remap(vertices.size());
size_t uniqueVertexCount = meshopt_generateVertexRemap(
    remap.data(),
    indices.data(),
    indices.size(),
    vertices.data(),
    vertices.size(),
    sizeof(Vertex));

// 2. Remap vertices and indices
std::vector<Vertex> remappedVertices(uniqueVertexCount);
meshopt_remapVertexBuffer(
    remappedVertices.data(),
    vertices.data(),
    vertices.size(),
    sizeof(Vertex),
    remap.data());

std::vector<unsigned int> remappedIndices(indices.size());
meshopt_remapIndexBuffer(
    remappedIndices.data(),
    indices.data(),
    indices.size(),
    remap.data());

// 3. Remove degenerate and duplicate triangles
std::vector<unsigned int> finalIndices(remappedIndices.size());
size_t finalCount = meshopt_filterIndexBuffer(
    finalIndices.data(),
    remappedIndices.data(),
    remappedIndices.size(),
    remappedVertices.data(),
    uniqueVertexCount,
    sizeof(float) * 3,
    sizeof(Vertex));

finalIndices.resize(finalCount);

Summary

• Degenerate triangles (zero-area) are removed by meshopt_filterIndexBuffer when two or more vertices in a triangle share identical position data.
• Duplicate triangles are eliminated based on vertex keys and winding order, preserving only the first occurrence.
• Opposite-winding duplicates are intentionally kept to support double-sided rendering requirements.
• The function compares only the first vertex_size bytes of each vertex, allowing you to filter by position while ignoring normals, UVs, or other attributes.
• Use meshopt_filterIndexBufferMulti when uniqueness depends on multiple vertex attributes or non-contiguous data.
• Always allocate the destination buffer to match the source size, then resize to the returned count after filtering.

Frequently Asked Questions

What is the difference between meshopt_filterIndexBuffer and meshopt_filterIndexBufferMulti?

meshopt_filterIndexBuffer compares a single contiguous block of bytes per vertex (typically the position), while meshopt_filterIndexBufferMulti accepts an array of meshopt_Stream structures to concatenate multiple attribute sources into a composite key. Use the multi-stream variant when your mesh requires matching bone weights, colors, or other attributes to determine true vertex uniqueness.

Does meshopt_filterIndexBuffer modify the original index buffer in place?

No, the function writes results to a separate destination buffer specified by the first parameter. The source index buffer remains unchanged, and the function returns the new index count to indicate how many elements were written to the destination.

Why does the function preserve triangles with opposite winding?

The implementation preserves duplicate triangles that have opposite winding order because they may be necessary for double-sided rendering or visual correctness in engines that do not rely on shader-sidedness. The filter specifically checks the vertex key order to distinguish between clockwise and counter-clockwise versions of the same geometric triangle.

How do I determine the correct vertex_size parameter for my mesh?

Set vertex_size to the number of bytes that constitute your position data—commonly sizeof(float) * 3 for 3D coordinates. If you need to include additional attributes in the equality check, either increase vertex_size to encompass those attributes (if they are contiguous with position) or switch to meshopt_filterIndexBufferMulti and define separate streams for each attribute group.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}