# Supported GGUF Quantization Layouts for Routed MoE Experts in DS4

> Discover the seven GGUF quantization layouts for routed MoE experts supported by DS4: Q8_0, IQ2_XXS, Q2_K, Q4_K, Q5_K, Q6_K, and MXFP4. Learn how DS4 ensures compatibility.

- Repository: [Salvatore Sanfilippo/ds4](https://github.com/antirez/ds4)
- Tags: deep-dive
- Published: 2026-08-07

---

**The DS4 engine explicitly supports seven GGUF quantization layouts for routed Mixture-of-Experts tensors: Q8_0, IQ2_XXS, Q2_K, Q4_K, Q5_K, Q6_K, and MXFP4, rejecting any other type at runtime via the `tensor_is_routed_expert_type` validator.**

The `antirez/ds4` repository implements a specialized inference engine for DeepSeek-style MoE models that enforces strict constraints on how routed expert weights are quantized. Unlike standard dense layers, the gate (*w₁*), up (*w₃*), and down (*w₂*) projection tensors for routed experts must use one of the predefined GGUF block formats to ensure compatibility with the engine's memory mapping and kernel dispatch logic.

## Complete List of Supported GGUF Layouts

The definitive enumeration appears in a block-format comment near the top of [`ds4.c`](https://github.com/antirez/ds4/blob/main/ds4.c) (lines 755-762) and is enforced by the validation logic in the same file. Each layout maps to a DS4 tensor constant and serves specific roles within the expert MLP:

- **Q8_0** (`DS4_TENSOR_Q8_0`): Standard 8-bit quantization used when memory is plentiful; suitable for any expert type.
- **IQ2_XXS** (`DS4_TENSOR_IQ2_XXS`): Ultra-low-bit integer quantization typically reserved for **gate (*w₁*)** and **up (*w₃*)** projectors.
- **Q2_K** (`DS4_TENSOR_Q2_K`): 2-bit "K-block" quantization optimized for **down (*w₂*)** projectors in routed experts.
- **Q4_K** (`DS4_TENSOR_Q4_K`): 4-bit block quantization balancing size and accuracy for general routed experts (high-memory variant).
- **Q5_K** (`DS4_TENSOR_Q5_K`): 5-bit block quantization used specifically by GLM-style routed experts.
- **Q6_K** (`DS4_TENSOR_Q6_K`): 6-bit block quantization used specifically by GLM-style routed experts.
- **MXFP4** (`DS4_TENSOR_MXFP4`): Mixed-precision format for **losslessly repacking** packed experts from native checkpoints.

## How DS4 Validates Expert Tensor Types

The runtime validation occurs in [`ds4.c`](https://github.com/antirez/ds4/blob/main/ds4.c) within the `tensor_is_routed_expert_type` function (around lines 4426-4434). This helper returns true only if the tensor's GGUF type matches one of the seven supported constants listed above.

If an unsupported layout is encountered, the `routed_expert_block_bytes` helper triggers a fatal error at lines 4444-4446 via its `default:` case, emitting an "unsupported routed expert tensor type" message and terminating execution. This strict validation prevents silent misinterpretation of quantized weights that could corrupt inference results.

## Quantizing Checkpoints with Specific Expert Layouts

The `deepseek4-quantize` CLI tool in [`gguf-tools/deepseek4-quantize.c`](https://github.com/antirez/ds4/blob/main/gguf-tools/deepseek4-quantize.c) exposes the supported layouts through dedicated flags. You can assign specific quantization schemes to each expert projection:

```bash

# Set gate (w1) and up (w3) to IQ2_XXS, down (w2) to Q4_K

deepseek4-quantize \
  --hf path/to/hf_dir \
  --template model_template.gguf \
  --out model_quantized.gguf \
  --routed-w1 iq2_xxs \
  --routed-w2 q4_k \
  --routed-w3 iq2_xxs \
  --n-experts 64

```

The `--routed-w1`, `--routed-w2`, and `--routed-w3` flags map directly to the GGUF layout constants. According to the usage text at lines 2650-2653 of [`deepseek4-quantize.c`](https://github.com/antirez/ds4/blob/main/deepseek4-quantize.c), these options accept the lowercase string identifiers shown above. Additionally, line 2664 documents **MXFP4** as a lossless repack option for preserving packed expert data from native checkpoints.

## Programmatic Validation

When working with the DS4 C API, you can verify tensor compatibility before inference:

```c
#include "ds4.h"

bool is_supported = tensor_is_routed_expert_type(tensor->type);
if (!is_supported) {
    ds4_die("unsupported routed expert tensor type");
}

```

This check mirrors the internal validation performed by the engine when loading GGUF files, ensuring your code fails fast if an incompatible expert tensor is encountered.

## Summary

- **Seven layouts supported**: Q8_0, IQ2_XXS, Q2_K, Q4_K, Q5_K, Q6_K, and MXFP4 are the only valid GGUF types for routed MoE experts in `antirez/ds4`.
- **Validation location**: The `tensor_is_routed_expert_type` function in [`ds4.c`](https://github.com/antirez/ds4/blob/main/ds4.c) (lines 4426-4434) enforces these constraints at runtime.
- **Typical assignments**: IQ2_XXS for gate/up projections, Q2_K or Q4_K for down projections, and MXFP4 for lossless repacking.
- **CLI tooling**: Use `deepseek4-quantize` with `--routed-w1`, `--routed-w2`, and `--routed-w3` flags to select layouts during quantization.
- **Error handling**: Unsupported types trigger a fatal error in `routed_expert_block_bytes` (lines 4444-4446 of [`ds4.c`](https://github.com/antirez/ds4/blob/main/ds4.c)).

## Frequently Asked Questions

### What happens if I use an unsupported GGUF quantization layout for routed experts?

The DS4 runtime will terminate with a fatal error. When loading a model, the `routed_expert_block_bytes` helper checks the tensor type, and if it does not match one of the seven supported layouts (Q8_0, IQ2_XXS, Q2_K, Q4_K, Q5_K, Q6_K, or MXFP4), the `default:` case at lines 4444-4446 of [`ds4.c`](https://github.com/antirez/ds4/blob/main/ds4.c) triggers an "unsupported routed expert tensor type" error.

### Which quantization layout should I use for gate versus down expert projections?

According to the DS4 source documentation, **IQ2_XXS** is optimized for **gate (*w₁*)** and **up (*w₃*)** projectors, while **Q2_K** is specifically designated for **down (*w₂*)** projectors. For higher accuracy at the cost of larger file sizes, **Q4_K** can be used for any expert type, and GLM-style models typically employ **Q5_K** or **Q6_K**.

### What is the MXFP4 layout used for?

**MXFP4** is a special mixed-precision format defined as `DS4_TENSOR_MXFP4` that performs **lossless repacking** of packed experts from native DeepSeek checkpoints. Unlike other layouts that re-quantize weights, MXFP4 preserves the original packed representation, making it ideal for converting pre-quantized models without precision loss. This is documented in [`deepseek4-quantize.c`](https://github.com/antirez/ds4/blob/main/deepseek4-quantize.c) at line 2664.

### Where are the supported quantization constants defined?

The string-to-constant mappings reside in [`gguf-tools/quants.h`](https://github.com/antirez/ds4/blob/main/gguf-tools/quants.h) under the `ds4q_type` enum, while the validation logic and layout descriptions live in [`ds4.c`](https://github.com/antirez/ds4/blob/main/ds4.c). The definitive comment block listing all supported GGUF block formats appears at lines 755-762 of [`ds4.c`](https://github.com/antirez/ds4/blob/main/ds4.c), and the runtime switch validating these types is implemented around lines 4426-4434 of the same file.