Llama.cpp Quantization Types: A Complete Guide to Accuracy and Performance Tradeoffs

Llama.cpp supports over a dozen quantization types that compress large language models from 32-bit floating point down to 2-bit integers, trading numerical precision for reduced memory footprint and faster inference.

Quantization types in llama.cpp determine how model weights are compressed and stored, directly impacting both the fidelity of generated text and the hardware resources required to run inference. The project implements these formats through two distinct enums: ggml_type defines the low-level tensor data types used by the GGML compute engine, while llama_ftype provides high-level model file flags that map specific quantization strategies to those underlying types.

Understanding Quantization Types in Llama.cpp

The quantization pipeline relies on definitions in two critical header files. In ggml/include/ggml.h (lines 389–430), the ggml_type enum declares every supported tensor format from full-precision GGML_TYPE_F32 down to 2-bit GGML_TYPE_IQ2_XXS. Meanwhile, include/llama.h (lines 116–155) defines llama_ftype, which assigns human-readable labels like LLAMA_FTYPE_MOSTLY_Q4_K to the underlying GGML types.

When you run the llama-quantize tool, the system converts raw FP16 or FP32 weights into these compressed formats using block-wise quantization, where small groups of weights share a single scale factor to minimize precision loss.

Quantization Format Taxonomy and Accuracy Tradeoffs

The following sections break down the major quantization families supported by llama.cpp, ordered from highest precision (and largest size) to most aggressive compression.

Full-Precision and 16-Bit Formats (F32, F16)

GGML_TYPE_F32 stores weights as 32-bit IEEE-754 floats, consuming 32 bits per weight and producing a 14.96 GiB model for Llama-3.1-8B. This format serves as the accuracy baseline with zero quantization loss but requires maximum memory bandwidth.

GGML_TYPE_F16 halves storage to 16 bits per weight (7.5 GiB for Llama-3.1-8B) with negligible perplexity increase. Performance remains comparable to F32, making F16 ideal when you have ample VRAM but want faster loading times.

4-Bit K-Quants (Q4_K, Q4_K_M)

GGML_TYPE_Q4_K and its mixture variant GGML_TYPE_Q4_K_M represent the project's recommended sweet spot, storing approximately 4.67 bits per weight (4.36–4.58 GiB). These formats use block-wise quantization with separate scales for weight magnitudes, yielding only ~0.2 perplexity point increase over F16 while achieving 71.9 tokens/second generation throughput on CPU.

5-Bit and 6-Bit K-Quants (Q5_K, Q6_K)

GGML_TYPE_Q5_K increases precision to 5.57 bits per weight (5.21 GiB), reducing perplexity loss to roughly 0.1 points below Q4_K levels while maintaining competitive 69.5 tokens/second generation speeds.

GGML_TYPE_Q6_K pushes to 6.56 bits per weight (6.14 GiB), offering a middle ground between F16 and Q5_K with noticeable quality improvements over 4-bit formats but slower 58.7 tokens/second throughput.

8-Bit Integer (Q8_0)

GGML_TYPE_Q8_0 stores 8.5 bits per weight (7.95 GiB), providing the highest fidelity among integer quantization formats. It achieves 865 tokens/second prompt processing speed, making it optimal for scenarios where you need near-F16 accuracy with slightly reduced memory footprint.

2-Bit and 3-Bit I-Quants (IQ2_XXS, IQ3_XXS)

GGML_TYPE_IQ2_XXS and related 2-bit formats compress to 2.0–2.6 bits per weight (1.87–2.56 GiB), enabling massive model compression. However, these i-quants (importance-matrix-guided quantization) require a pre-generated imatrix file to identify critical weights. With a proper importance matrix, IQ2 formats can match Q4_K quality; without it, perplexity degradation becomes severe.

GGML_TYPE_IQ3_XXS through GGML_TYPE_IQ3_S offer 3.25–3.5 bits per weight (3.04–3.27 GiB), providing a balanced compression ratio when paired with importance matrix data.

Legacy and Specialized Formats (Q4_0, Q4_1, MXFP4)

GGML_TYPE_Q4_0 and GGML_TYPE_Q4_1 represent the original 4-bit quantization implementation, storing exactly 4 bits per weight (4.36 GiB). While simpler and well-tested, they generally underperform the newer Q4_K variants in accuracy-per-bit efficiency.

GGML_TYPE_MXFP4 implements the Microscaling FP4 format optimized for Mixture-of-Experts (MoE) architectures, storing 4.68 bits per weight (4.38 GiB). It delivers lower inference latency on MoE models compared to standard k-quants while maintaining comparable numerical fidelity to Q4_K.

How Llama.cpp Selects Quantization Types

The quantization process relies on the llama_tensor_get_type function defined in src/llama-quant.cpp. When llama-quantize executes with params.pure set to false (the default), this function applies sophisticated heuristics to assign optimal types per tensor:

• Layer-aware rules: Early and late layers often receive higher-bit types through use_more_bits logic to preserve embedding quality and output coherence.
• Tensor-name patterns: Specific weight categories like attn_v.weight, ffn_down, and token_embd.weight trigger automatic quality bumps based on their sensitivity to quantization error.
• Model-specific quirks: Falcon, Mixtral, and 70B-class models receive specialized handling to account for their unique architectural characteristics.
• Imbalance checks: When the column count (nx) isn't a multiple of the target quantization's block size, the code falls back to higher-bit types (lines 780–805 of llama-quant.cpp) to prevent alignment errors.

The final selected type is written to the GGUF output file via gguf_set_tensor_type (lines 530–555).

Generating Importance Matrices for Low-Bit Quantization

Aggressive compression formats like IQ2_XXS, IQ3_XXS, and MXFP4 require an importance matrix (imatrix) to achieve acceptable accuracy. This matrix identifies which weights contribute most to model output, allowing the quantizer to allocate precision where it matters most.

Without a valid imatrix, llama-quantize aborts when processing these formats (lines 778–785 of src/llama-quant.cpp).

Generate the importance matrix using the dedicated tool:

./llama-imatrix \
    -m ./models/ggml-model-f16.gguf \
    -o imatrix.gguf \
    -f calibration.txt

The calibration file should contain representative text samples from your target domain. For faster processing on CUDA-enabled systems, GPU-accelerated imatrix generation is available.

Quantizing a Model with Llama.cpp

Convert a full-precision model to a quantized format using the llama-quantize CLI. This example converts to Q4_K_M with importance matrix guidance and custom tensor type overrides:

# Generate importance matrix first (recommended for best accuracy)

./llama-imatrix -m ./models/ggml-model-f16.gguf -o imatrix.gguf

# Quantize with specific tensor type overrides

./llama-quantize \
    --imatrix imatrix.gguf \
    --output-tensor-type f16 \
    --token-embedding-type q3_k \
    ./models/ggml-model-f16.gguf \
    ./models/ggml-model-Q4_K_M.gguf \
    q4_k_m \
    8

Parameter breakdown:

• q4_k_m: Selects the Q4_K mixture format (maps to GGML_TYPE_Q4_K)
• 8: Thread count for parallel quantization
• --output-tensor-type f16: Keeps the final output layer in 16-bit float for quality
• --token-embedding-type q3_k: Uses 3-bit quantization for the token embedding table to save memory

Running Inference on Quantized Models

Load and run quantized models using the standard CLI interface. Modern GGUF files are automatically recognized, though the -cnv flag handles legacy format conversions if needed:

./llama-cli \
    -m ./models/ggml-model-Q4_K_M.gguf \
    -cnv \
    -p "Explain the tradeoffs between quantization formats"

The -cnv flag enables automatic conversion of legacy quantization formats to the internal representation used by the compute backend.

Key Source Files for Quantization Types

File	Purpose
[ggml/include/ggml.h](https://github.com/ggml-org/llama.cpp/blob/master/ggml/include/ggml.h)	Defines the ggml_type enum and helper functions (ggml_type_name, ggml_blck_size) for all tensor formats.
[include/llama.h](https://github.com/ggml-org/llama.cpp/blob/master/include/llama.h)	Declares llama_ftype enum mapping high-level file types (e.g., LLAMA_FTYPE_MOSTLY_Q4_K) to underlying GGML types.
[src/llama-quant.cpp](https://github.com/ggml-org/llama.cpp/blob/master/src/llama-quant.cpp)	Implements llama_tensor_get_type with layer-aware heuristics, tensor pattern matching, and fallback logic for block-size alignment.
[tools/quantize/quantize.cpp](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/quantize.cpp)	CLI entry point parsing arguments and invoking llama_model_quantize.
[tools/imatrix/imatrix.cpp](https://github.com/ggml-org/llama.cpp/blob/master/tools/imatrix/imatrix.cpp)	Generates importance matrices required for 2-bit and 3-bit i-quant formats.
[examples/llama-cli.cpp](https://github.com/ggml-org/llama.cpp/blob/master/examples/llama-cli.cpp)	Demonstrates runtime loading and inference with quantized GGUF models.

Summary

• Higher-bit formats (F16, Q8_0, Q6_K) preserve near-baseline accuracy but offer modest compression (2× to 4× reduction).
• 4-bit k-quants (Q4_K, Q4_K_M) represent the optimal balance for most deployments, achieving 4–5× compression with minimal perplexity increase (~0.2 points) and high generation throughput (~72 tokens/second).
• 5-bit k-quants (Q5_K) provide a middle ground between Q4_K and F16, trading 20% larger size for improved numerical fidelity.
• Low-bit i-quants (IQ2_XXS, IQ3_XXS) enable extreme compression (2–3 bits per weight) but require an importance matrix to maintain quality; without it, accuracy degrades significantly.
• Specialized formats like MXFP4 optimize inference latency for Mixture-of-Experts architectures.

Frequently Asked Questions

What is the difference between ggml_type and llama_ftype in llama.cpp?

ggml_type is the low-level tensor data type enum defined in ggml/include/ggml.h that specifies how individual weight blocks are stored (e.g., GGML_TYPE_Q4_K). llama_ftype is the high-level file type enum defined in include/llama.h that maps user-facing quantization names (e.g., LLAMA_FTYPE_MOSTLY_Q4_K) to the underlying GGML types used throughout the model.

Why do 2-bit and 3-bit quantization types require an importance matrix?

Aggressive compression formats like IQ2_XXS and IQ3_XXS discard significant numerical precision, so they rely on an importance matrix (imatrix) to identify which weights most influence model output. The quantizer uses this data to allocate higher effective precision to critical parameters. Without the imatrix, as enforced in src/llama-quant.cpp (lines 778–785), these formats produce unacceptable quality degradation.

Which quantization type offers the best balance between model size and accuracy?

For most production deployments, Q4_K_M (4-bit k-quant mixture) provides the optimal tradeoff. It compresses an 8B parameter model to approximately 4.6 GiB (4.67 bits per weight) while increasing perplexity by only ~0.2 points compared to FP16. It also maintains high generation throughput (~72 tokens/second), making it suitable for both consumer GPUs and high-performance CPU inference.

How does llama.cpp automatically select quantization types for different layers?

The llama_tensor_get_type function in src/llama-quant.cpp applies layer-aware heuristics when params.pure is false. It examines tensor names (e.g., attn_v.weight, ffn_down) to apply use_more_bits logic for sensitive layers, checks for model-specific quirks (Falcon, Mixtral, 70B architectures), and validates block-size alignment (falling back to higher-bit types when nx isn't divisible by the target block size, as seen in lines 780–805).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}