Preprocessing Requirements for meshoptimizer Index Buffer Compression

meshoptimizer index buffer compression requires the input index buffer to be preprocessed through vertex remapping, cache optimization, and fetch optimization to achieve the target ratio of approximately one byte per triangle; skipping these steps causes the encoder to fall back to varint encoding, producing significantly larger outputs.

The meshopt_encodeIndexBuffer and meshopt_decodeIndexBuffer functions in the zeux/meshoptimizer repository assume specific locality patterns in the input data. The codec relies on a 16-entry edge FIFO and a 16-entry vertex FIFO to encode triangle indices efficiently, and these data structures only hit their fast paths when the mesh has been optimized for GPU cache coherency. Without proper preprocessing, the encoder not only produces poor compression ratios but may also generate data that the decoder cannot reconstruct correctly.

Step 1: Generate a Proper Indexed Mesh

Before compression, you must convert raw vertex streams into a compact indexed format using vertex remapping. Duplicate vertices waste space and break the FIFO-based delta coding assumptions hardcoded in the encoder.

Call meshopt_generateVertexRemap (implemented in src/vertexcodec.cpp) to build a remap table that eliminates duplicate vertices. This function creates a dense mapping between your raw vertex stream and a compact vertex buffer where each unique attribute combination appears exactly once.

std::vector<unsigned int> remap(vertexCount);
size_t newVertexCount = meshopt_generateVertexRemap(
    remap.data(),
    nullptr,                    // No original index buffer
    vertexCount,
    verts.data(),
    vertexCount,
    sizeof(Vertex));

After generating the remap table, apply it to both your indices and vertices using meshopt_remapIndexBuffer and meshopt_remapVertexBuffer. The resulting compact index buffer is the prerequisite for all subsequent optimization steps.

Step 2: Optimize Vertex Cache Locality

Run meshopt_optimizeVertexCache (implemented in src/vcacheoptimizer.cpp at line 35) to reorder triangles for maximum GPU cache coherency. This step is critical because the encoder’s primary fast path looks for repeated edges in its 16-entry edge FIFO.

When triangles share edges with recent triangles in the sequence, the encoder emits a single byte containing the edge FIFO index (fe) and a vertex FIFO index (fec). Cache-friendly ordering maximizes these edge FIFO hits, keeping the per-triangle cost at the theoretical minimum of 1 byte. Without this optimization, the encoder falls back to expensive variable-length integer encoding for most triangles.

If your rendering pipeline can tolerate a potential change of the provoking vertex, use meshopt_optimizeVertexCacheStrip (line 70 in src/vcacheoptimizer.cpp) instead. This variant can produce even more edge FIFO matches, pushing the average size closer to the 1 byte/triangle bound.

Step 3: Optimize Vertex Fetch Order

Apply meshopt_optimizeVertexFetch (implemented in src/vfetchoptimizer.cpp at line 30) after cache optimization. This function reorders the vertex buffer so that vertices appear in the order they are referenced by the index buffer.

The codec maintains a 16-entry vertex FIFO; when a vertex index appears in this cache, the encoder emits a short "fec" code instead of a full varint. Fetch-friendly ordering makes vertex FIFO hits frequent, dramatically reducing the bitstream size. This step must occur after cache optimization because reordering vertices changes the locality patterns that the cache optimizer established.

Step 4: Zero-Initialize Padding Bytes

Ensure any padding bytes in your vertex struct are set to zero before encoding. The vertex codec treats the entire vertex stride as payload data; non-zero padding would be treated as data to encode, inflating the compressed size unnecessarily.

While this step applies to meshopt_encodeVertexBuffer (defined in src/vertexcodec.cpp at line 104), it indirectly affects index compression quality because the optimized index buffer references the vertex stream. Clean vertex data ensures the encoder can focus entirely on index locality patterns.

Complete Preprocessing Pipeline

The following C++ example demonstrates the full preprocessing pipeline required before calling meshopt_encodeIndexBuffer:

// Generate compact index buffer from raw data
std::vector<unsigned int> remap(faceCount * 3);
size_t newVertexCount = meshopt_generateVertexRemap(
    remap.data(),
    nullptr,
    faceCount * 3,
    vertices.data(),
    faceCount * 3,
    sizeof(Vertex));

std::vector<unsigned int> indices(faceCount * 3);
std::vector<Vertex> compactVerts(newVertexCount);

meshopt_remapIndexBuffer(indices.data(), nullptr, indices.size(), remap.data());
meshopt_remapVertexBuffer(
    compactVerts.data(),
    vertices.data(),
    faceCount * 3,
    sizeof(Vertex),
    remap.data());

// Optimize for vertex cache locality (16-entry edge FIFO)
meshopt_optimizeVertexCache(
    indices.data(),
    indices.data(),
    indices.size(),
    newVertexCount);

// Optimize for vertex fetch (16-entry vertex FIFO)
meshopt_optimizeVertexFetch(
    compactVerts.data(),
    indices.data(),
    indices.size(),
    compactVerts.data(),
    newVertexCount,
    sizeof(Vertex));

// Encode the optimized index buffer
std::vector<unsigned char> compressed(
    meshopt_encodeIndexBufferBound(indices.size(), newVertexCount));
size_t compressedSize = meshopt_encodeIndexBuffer(
    compressed.data(),
    compressed.size(),
    indices.data(),
    indices.size());

How the Encoder Leverages Optimization

The index codec (implemented in src/indexcodec.cpp) processes triangles sequentially using two key data structures:

• 16-entry Edge FIFO: Tracks recently seen edges. When a triangle shares an edge with a recent triangle, the encoder emits a single byte containing the edge index (fe) and a vertex FIFO reference (fec).
• 16-entry Vertex FIFO: Caches recently referenced vertex indices. Hits emit short "fec" codes instead of full varint-encoded indices.

The encoder writes a versioned header byte, then iterates through triangles. If an edge match is found in the FIFO, it takes the "edge-FIFO path" costing 1 byte. If no edge match exists but a vertex match exists, it emits a slightly larger code. Only new vertices require full varint encoding.

Because the fast path requires fe < 15 and fec < fecmax, the preprocessing steps directly determine compression ratio. As documented in the repository's README.md, "index encoding assumes that the index buffer was optimized for vertex cache and vertex fetch."

Summary

• Vertex remapping is required to eliminate duplicate vertices and create a compact index buffer using meshopt_generateVertexRemap from src/vertexcodec.cpp.
• Cache optimization via meshopt_optimizeVertexCache (in src/vcacheoptimizer.cpp) maximizes 16-entry edge FIFO hits, enabling the 1-byte-per-triangle fast path.
• Fetch optimization via meshopt_optimizeVertexFetch (in src/vfetchoptimizer.cpp) maximizes 16-entry vertex FIFO hits, reducing fallback to varint encoding.
• Zero padding in vertex structures prevents the vertex codec from encoding garbage data as payload.
• The meshopt_encodeIndexBuffer function in src/indexcodec.cpp assumes these optimizations have been applied; unoptimized input produces poor compression or incorrect decoding.

Frequently Asked Questions

What happens if I skip vertex cache optimization before encoding?

Without vertex cache optimization, the edge FIFO (as implemented in src/indexcodec.cpp) rarely finds matching edges. The encoder falls back to emitting full variable-length integers for most triangle indices, typically increasing the compressed size by 3-4x compared to the optimized 1 byte per triangle target.

Is vertex fetch optimization required for correctness or just compression?

Vertex fetch optimization is required strictly for compression ratio, not correctness. The decoder will produce valid indices without it, but the encoded stream will be significantly larger because the vertex FIFO (used by the encoder to emit "fec" short codes) will miss frequently, forcing expensive full-index encodings.

Can I use meshopt_optimizeVertexCacheStrip instead of the standard optimizer?

Yes. meshopt_optimizeVertexCacheStrip (defined in src/vcacheoptimizer.cpp at line 70) can produce even better compression ratios when you can tolerate potential changes to the provoking vertex. This variant generates strip-like orderings that increase edge FIFO reuse, though the standard meshopt_optimizeVertexCache is sufficient for most use cases.

How does vertex quantization affect index buffer compression?

While quantization (using helpers like meshopt_quantizeSnorm) is not strictly required for index encoding correctness, it reduces vertex data entropy. Lower entropy in the vertex stream often correlates with better coherency in the index stream, allowing the edge and vertex FIFOs in src/indexcodec.cpp to achieve higher hit rates and thus better compression.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}