How to Use Vertex Filters for glTF Serialization Without Shader Changes

Meshoptimizer's vertex filter functions encode attributes into compact representations before compression, storing the filter name in the EXT_meshopt_compression extension so that decoders restore original values automatically without requiring shader modifications.

The zeux/meshoptimizer library provides specialized vertex filter functions that enable aggressive compression of glTF assets while maintaining complete backward compatibility with existing shaders. By encoding attributes like normals and tangents into compact formats such as octahedral or quaternion representations before compression, you can significantly reduce file size without modifying your GPU code. This approach leverages the EXT_meshopt_compression extension to store both the compressed data and the filter metadata, ensuring transparent decoding at runtime.

How the Vertex Filter Pipeline Works

The vertex filter pipeline in meshoptimizer operates as a preprocessing step that transforms floating-point attributes into quantized representations optimized for entropy coding. According to the source code in src/vertexfilter.cpp, the library implements several encoding functions including meshopt_encodeFilterOct, meshopt_encodeFilterQuat, meshopt_encodeFilterColor, and meshopt_encodeFilterExp, each designed for specific attribute types.

The process follows five distinct stages from export to rendering:

Step 1: Encode Attributes with Filter Functions

Raw float attribute data passes through one of the encode filter functions located in src/vertexfilter.cpp. For example, meshopt_encodeFilterOct transforms unit vectors into octahedral-encoded representations, while meshopt_encodeFilterQuat compresses quaternion rotations. These functions write filtered buffer data into a destination array using specified bit depths, typically 8 to 16 bits per component.

Step 2: Compress the Filtered Buffer

The filtered buffer feeds into the vertex buffer encoder implemented in src/vertexcodec.cpp. The meshopt_encodeVertexBuffer function compresses the already-filtered data, producing a payload stored in a glTF buffer view. This two-stage compression (filter then encode) achieves higher compression ratios than either step alone.

Step 3: Write Filter Metadata to glTF

During serialization, gltf/write.cpp records the filter name in the EXT_meshopt_compression extension of the accessor. The extension stores values like "filter":"OCT" or "filter":"QUAT" alongside the compressed bytes. This metadata tells the loader which inverse transform to apply after decompression.

Step 4: Decode at Runtime

When loading the asset, the decoder calls the corresponding meshopt_decodeFilter* functions from src/vertexfilter.cpp automatically. These routines restore the attribute to the exact layout expected by the shader, reversing the octahedral, quaternion, or exponential encoding applied during export.

Step 5: Shader Transparency

Because the decoder restores the original data layout and value ranges, the shader receives conventional attribute semantics. A NORMAL attribute remains a unit vec3, and a TANGENT remains a vec4, requiring no modification to vertex shader code.

Encoding Vertex Attributes for glTF Export

To implement vertex filtering in your export pipeline, you encode the attributes before compression and ensure the glTF writer records the filter type. The following C++ example demonstrates encoding normals with octahedral filtering using 8 bits per component:

// Prepare source data: vertex normals as float4 (xyz + padding)
size_t vertexCount = mesh.vertexCount;
std::vector<float> normals(vertexCount * 4);
std::vector<short> filtered(vertexCount * 4);  // Output buffer

// Encode with octahedral filter (8 bits per component)
meshopt_encodeFilterOct(
    filtered.data(),          // Destination buffer
    vertexCount,              // Number of vertices
    4,                        // Stride (4 components)
    8,                        // Bits per component (2..16)
    normals.data()            // Source float data
);

// Compress the filtered buffer
size_t maxSize = meshopt_encodeVertexBufferBound(filtered.size() * sizeof(short), 0);
std::vector<unsigned char> compressed(maxSize);

size_t compressedSize = meshopt_encodeVertexBuffer(
    compressed.data(),
    filtered.data(),
    filtered.size() * sizeof(short),
    4,                        // Stride in bytes (4 bytes = 2 components × 16 bits)
    0                         // No special options
);
compressed.resize(compressedSize);

// Export with gltfpack - automatically writes filter metadata
// Command: gltfpack -i input.gltf -o output.glb --filter-oct

When using gltfpack from gltf/gltfpack.cpp, specify the filter type via command-line flags like --filter-oct, --filter-quat, or --filter-exp. The tool automatically handles the encoding step and writes the correct filter name to the EXT_meshopt_compression extension.

Loading Filtered Assets in JavaScript

At runtime, the meshoptimizer decoder handles filter inversion transparently after decompression. The JavaScript loader initializes the decoder, which automatically applies the appropriate meshopt_decodeFilter* function based on the metadata stored in the glTF extension:

// Initialize the decoder
await MeshoptDecoder.ready;

// Load the glTF asset
const gltf = await loadGLTF('output.glb');

// After buffer decompression, the decoder automatically runs:
// meshopt_decodeFilterOct(buffer, vertexCount, 4);
// This restores the original float normals before the shader reads them

No additional JavaScript code is required to handle the filtering. The decoder reads the filter name from the accessor's EXT_meshopt_compression extension and applies the inverse transform, presenting the same data layout as an uncompressed glTF file.

Supported Filter Types and Bit Depths

The src/vertexfilter.cpp implementation supports four primary filter modes, each optimized for specific attribute types:

• meshopt_encodeFilterOct: Encodes unit vectors (normals, tangents) using octahedral mapping. Supports 2 to 16 bits per component.
• meshopt_encodeFilterQuat: Compresses quaternion rotations into 64-bit or 128-bit representations.
• meshopt_encodeFilterColor: Optimizes vertex colors using exponential filtering for HDR or LDR data.
• meshopt_encodeFilterExp: General-purpose exponential quantization for scalar attributes.

Each function preserves the semantic meaning of the data while reducing entropy, enabling the subsequent vertex codec to achieve higher compression ratios.

Key Source Files and Functions

Understanding the implementation requires familiarity with these specific files in the zeux/meshoptimizer repository:

• src/vertexfilter.cpp: Contains the encode functions (meshopt_encodeFilterOct, meshopt_encodeFilterQuat, meshopt_encodeFilterColor, meshopt_encodeFilterExp) and decode functions (meshopt_decodeFilterOct, etc.).
• src/vertexcodec.cpp: Implements meshopt_encodeVertexBuffer and meshopt_encodeVertexBufferBound for compressing filtered buffers.
• gltf/write.cpp: Serializes glTF files and writes the EXT_meshopt_compression extension with filter metadata.
• gltf/gltfpack.cpp: Command-line interface that orchestrates the complete pipeline, accepting flags like --filter-oct and --filter-quat.

Summary

• Vertex filters in meshoptimizer transform attributes into compact representations before compression, reducing entropy and improving compression ratios.
• The EXT_meshopt_compression extension stores the filter name (e.g., "OCT", "QUAT") alongside compressed data, enabling automatic decoding.
• No shader changes are required because decode functions restore the original data layout and ranges before the GPU reads the attributes.
• Filter functions reside in src/vertexfilter.cpp, while compression logic lives in src/vertexcodec.cpp.
• gltfpack automates the pipeline via command-line flags, writing appropriate metadata to the glTF file.

Frequently Asked Questions

What vertex attribute types support filtering in meshoptimizer?

The library supports filtering for unit vectors via meshopt_encodeFilterOct (normals, tangents), rotations via meshopt_encodeFilterQuat (tangent frames), colors via meshopt_encodeFilterColor, and general scalars via meshopt_encodeFilterExp. Each function in src/vertexfilter.cpp targets specific data distributions to minimize quantization error while maximizing compression.

Do I need to modify my glTF loader to use vertex filters?

No modifications are necessary if your loader already supports EXT_meshopt_compression. The meshoptimizer decoder automatically detects the filter name stored in the extension and calls the appropriate meshopt_decodeFilter* function. The decoded buffer presents the same layout as uncompressed data, requiring no changes to how your application processes the glTF accessors.

How does the octahedral filter affect precision compared to raw floating-point storage?

The meshopt_encodeFilterOct function maps unit vectors to a square domain using octahedral projection, then quantizes to 8-16 bits per component. At 8 bits, the maximum angular error is approximately 0.5 degrees, while 12 bits provides error below 0.02 degrees. This precision range suits most real-time rendering applications while reducing storage by 50-75% compared to float32.

Can I apply different filters to different attributes in the same glTF file?

Yes. The EXT_meshopt_compression extension records filter types per-accessor, allowing you to encode normals with octahedral filtering while using exponential quantization for texture coordinates or quaternion filtering for tangent frames. The gltf/write.cpp implementation writes distinct filter metadata for each compressed accessor independently.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}