What Quantization Methods Are Supported by LlamaFactory? A Complete Guide to 9 Techniques

LlamaFactory supports nine quantization methods including BNB (BitsAndBytes), GPTQ, AWQ, AQLM, EETQ, HQQ, MXFP4, FP8, and QUANTO, covering both runtime (on-the-fly) and post-training quantization scenarios.

LlamaFactory provides a unified quantization interface that handles both post-training quantization (PTQ) and runtime quantization through a centralized configuration system. All supported methods are defined in the QuantizationMethod enum and integrated into the model initialization logic, allowing users to compress models efficiently for inference and training.

Overview of LlamaFactory Quantization Methods

The supported quantization methods are declared in src/llamafactory/extras/constants.py (lines 135-147) within the QuantizationMethod enum. These methods fall into three distinct categories based on when and how quantization is applied.

On-the-Fly (Runtime) Quantization Methods

These techniques quantize models during loading without requiring pre-quantized checkpoints:

• BNB (BitsAndBytes) – Supports 4-bit and 8-bit quantization via the bitsandbytes library
• EETQ – Provides efficient 8-bit quantization using EetqConfig
• HQQ – Enables flexible 1-8 bit quantization through the hqq library

Post-Training Quantization (PTQ) Methods

These methods require exporting models to quantized formats before deployment:

• GPTQ – Supports 2/3/4/8-bit export quantization via optimum.gptq
• AWQ – Activation-aware weight quantization requiring the autoawq dependency
• AQLM – Advanced quantization with default 2-bit precision
• MXFP4 – Mixed-precision FP4 quantization using Mxfp4Config
• FP8 – Fine-grained FP8 quantization via FineGrainedFP8Config

Future and Experimental Support

• QUANTO – Reserved enum entry for upcoming quanto integration (currently a placeholder without active pipeline wiring)

How Each Quantization Method Works in LlamaFactory

LlamaFactory implements method-specific configurations in src/llamafactory/model/model_utils/quantization.py. The configure_quantization() function (lines 82-95) orchestrates the initialization logic for each technique.

BNB (BitsAndBytes)

BNB provides dynamic 4-bit and 8-bit quantization during model loading. In quantization.py (lines 167-182), the system instantiates BitsAndBytesConfig when quantization_method is set to BNB.

# Example configuration triggering BNB 4-bit quantization

model_args = {
    "quantization_bit": 4,
    "quantization_method": "bnb"
}

GPTQ

GPTQ handles both export-time quantization and runtime inference for pre-quantized models. The implementation uses GPTQConfig (lines 332-360) when export_quantization_bit is specified or when loading GPTQ-formatted checkpoints.

# Export-time GPTQ configuration

model_args = {
    "export_quantization_bit": 4,
    "quantization_method": "gptq"
}

AWQ

AWQ support triggers the autoawq dependency when detected. In configure_quantization() (lines 122-124), the code checks for QuantizationMethod.AWQ and applies the appropriate configuration for activation-aware weight quantization.

AQLM

AQLM (Additive Quantization of Language Models) defaults to 2-bit precision. When quant_method == QuantizationMethod.AQLM (lines 125-128), the system explicitly sets bits = 2 for the AQLM quantizer configuration.

EETQ

EETQ provides efficient 8-bit quantization through EetqConfig (lines 207-217). This method is instantiated when the quantization_method enum resolves to EETQ.

HQQ

HQQ (Half-Quadratic Quantization) supports bit widths from 1 to 8 bits. The configuration is handled via HqqConfig (lines 195-207), offering granular control over quantization precision.

MXFP4 and FP8

These mixed-precision formats use dedicated configuration classes:

• MXFP4: Configured via Mxfp4Config (lines 103-108) when quant_method == QuantizationMethod.MXFP4
• FP8: Uses FineGrainedFP8Config (lines 110-115) when quant_method == QuantizationMethod.FP8

How LlamaFactory Selects and Applies Quantization

The quantization pipeline in configure_quantization() follows three distinct resolution paths:

1. Explicit PTQ Export – When model_args.export_quantization_bit is set, LlamaFactory uses GPTQ via optimum.gptq to perform 2/3/4/8-bit export quantization.

2. Pre-Quantized Checkpoints – If a loaded checkpoint contains a quantization_config, the system inspects the quant_method field (supporting MXFP4, FP8, GPTQ, AWQ, or AQLM) and applies the corresponding configuration automatically.

3. Runtime Quantization – When model_args.quantization_bit is provided at runtime, the specified quantization_method (BNB, HQQ, or EETQ) determines which quantization library is instantiated.

Configuring Quantization in Practice

To apply quantization in LlamaFactory, specify the method and bit precision through model arguments or training configuration files.

# Runtime quantization with BNB 4-bit

from llamafactory.extras.constants import QuantizationMethod

model_args = {
    "quantization_bit": 4,
    "quantization_method": QuantizationMethod.BNB
}

# PTQ export with GPTQ

export_args = {
    "export_quantization_bit": 4,
    "quantization_method": QuantizationMethod.GPTQ
}

For pre-quantized model loading, LlamaFactory automatically detects the quantization method from the model's config.json and applies the appropriate handler in configure_quantization().

Summary

• LlamaFactory supports nine quantization methods: BNB, GPTQ, AWQ, AQLM, EETQ, HQQ, MXFP4, FP8, and QUANTO (placeholder)
• Runtime methods (BNB, EETQ, HQQ) quantize models on-the-fly during loading
• PTQ methods (GPTQ, AWQ, AQLM, MXFP4, FP8) require export or pre-quantized checkpoints
• Method selection logic is centralized in configure_quantization() in src/llamafactory/model/model_utils/quantization.py
• The QuantizationMethod enum in src/llamafactory/extras/constants.py (lines 135-147) defines all supported techniques

Frequently Asked Questions

What is the difference between BNB and GPTQ in LlamaFactory?

BNB (BitsAndBytes) performs quantization at runtime when loading the model, requiring no pre-processing and supporting 4-bit and 8-bit formats. GPTQ is primarily used for post-training quantization exports (2/3/4/8-bit) or loading pre-quantized GPTQ models, offering higher compression ratios but requiring the optimum library and export time processing.

Can I use AWQ quantization for training in LlamaFactory?

AWQ in LlamaFactory is designed for inference and export scenarios using pre-quantized models or the autoawq library. While you can load AWQ-quantized models for fine-tuning with adapters, the quantization itself is applied post-training or via pre-quantized checkpoints, not during the training forward pass.

Which LlamaFactory quantization method offers the best memory efficiency?

AQLM provides the highest compression with 2-bit quantization by default (configurable in quantization.py lines 125-128), followed by HQQ which supports bit widths as low as 1-bit. For runtime quantization without pre-processing, BNB 4-bit (NF4) offers excellent memory reduction with minimal accuracy loss.

How do I check if my model is using the correct quantization method?

LlamaFactory validates quantization configuration in configure_quantization() (lines 82-95). Check the model's quantization_config in the generated config files, or inspect the logs during initialization where the library reports which QuantizationMethod enum value is active and the corresponding config class being instantiated (e.g., BitsAndBytesConfig, EetqConfig, or HqqConfig).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}