# What Quantization Methods Are Supported by LlamaFactory? A Complete Guide to 9 Techniques

> Explore LlamaFactory's comprehensive support for nine quantization methods like BNB GPTQ AWQ and more. Discover runtime and post-training techniques for efficient model deployment.

- Repository: [Yaowei Zheng/LlamaFactory](https://github.com/hiyouga/LlamaFactory)
- Tags: how-to-guide
- Published: 2026-03-04

---

**LlamaFactory supports nine quantization methods including BNB (BitsAndBytes), GPTQ, AWQ, AQLM, EETQ, HQQ, MXFP4, FP8, and QUANTO, covering both runtime (on-the-fly) and post-training quantization scenarios.**

LlamaFactory provides a unified quantization interface that handles both post-training quantization (PTQ) and runtime quantization through a centralized configuration system. All supported methods are defined in the `QuantizationMethod` enum and integrated into the model initialization logic, allowing users to compress models efficiently for inference and training.

## Overview of LlamaFactory Quantization Methods

The supported quantization methods are declared in [`src/llamafactory/extras/constants.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/extras/constants.py) (lines 135-147) within the `QuantizationMethod` enum. These methods fall into three distinct categories based on when and how quantization is applied.

### On-the-Fly (Runtime) Quantization Methods

These techniques quantize models during loading without requiring pre-quantized checkpoints:

- **BNB (BitsAndBytes)** – Supports 4-bit and 8-bit quantization via the `bitsandbytes` library
- **EETQ** – Provides efficient 8-bit quantization using `EetqConfig`
- **HQQ** – Enables flexible 1-8 bit quantization through the `hqq` library

### Post-Training Quantization (PTQ) Methods

These methods require exporting models to quantized formats before deployment:

- **GPTQ** – Supports 2/3/4/8-bit export quantization via `optimum.gptq`
- **AWQ** – Activation-aware weight quantization requiring the `autoawq` dependency
- **AQLM** – Advanced quantization with default 2-bit precision
- **MXFP4** – Mixed-precision FP4 quantization using `Mxfp4Config`
- **FP8** – Fine-grained FP8 quantization via `FineGrainedFP8Config`

### Future and Experimental Support

- **QUANTO** – Reserved enum entry for upcoming `quanto` integration (currently a placeholder without active pipeline wiring)

## How Each Quantization Method Works in LlamaFactory

LlamaFactory implements method-specific configurations in [`src/llamafactory/model/model_utils/quantization.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/model/model_utils/quantization.py). The `configure_quantization()` function (lines 82-95) orchestrates the initialization logic for each technique.

### BNB (BitsAndBytes)

BNB provides dynamic 4-bit and 8-bit quantization during model loading. In [`quantization.py`](https://github.com/hiyouga/LlamaFactory/blob/main/quantization.py) (lines 167-182), the system instantiates `BitsAndBytesConfig` when `quantization_method` is set to `BNB`.

```python

# Example configuration triggering BNB 4-bit quantization

model_args = {
    "quantization_bit": 4,
    "quantization_method": "bnb"
}

```

### GPTQ

GPTQ handles both export-time quantization and runtime inference for pre-quantized models. The implementation uses `GPTQConfig` (lines 332-360) when `export_quantization_bit` is specified or when loading GPTQ-formatted checkpoints.

```python

# Export-time GPTQ configuration

model_args = {
    "export_quantization_bit": 4,
    "quantization_method": "gptq"
}

```

### AWQ

AWQ support triggers the `autoawq` dependency when detected. In `configure_quantization()` (lines 122-124), the code checks for `QuantizationMethod.AWQ` and applies the appropriate configuration for activation-aware weight quantization.

### AQLM

AQLM (Additive Quantization of Language Models) defaults to 2-bit precision. When `quant_method == QuantizationMethod.AQLM` (lines 125-128), the system explicitly sets `bits = 2` for the AQLM quantizer configuration.

### EETQ

EETQ provides efficient 8-bit quantization through `EetqConfig` (lines 207-217). This method is instantiated when the `quantization_method` enum resolves to `EETQ`.

### HQQ

HQQ (Half-Quadratic Quantization) supports bit widths from 1 to 8 bits. The configuration is handled via `HqqConfig` (lines 195-207), offering granular control over quantization precision.

### MXFP4 and FP8

These mixed-precision formats use dedicated configuration classes:

- **MXFP4**: Configured via `Mxfp4Config` (lines 103-108) when `quant_method == QuantizationMethod.MXFP4`
- **FP8**: Uses `FineGrainedFP8Config` (lines 110-115) when `quant_method == QuantizationMethod.FP8`

## How LlamaFactory Selects and Applies Quantization

The quantization pipeline in `configure_quantization()` follows three distinct resolution paths:

1. **Explicit PTQ Export** – When `model_args.export_quantization_bit` is set, LlamaFactory uses **GPTQ** via `optimum.gptq` to perform 2/3/4/8-bit export quantization.

2. **Pre-Quantized Checkpoints** – If a loaded checkpoint contains a `quantization_config`, the system inspects the `quant_method` field (supporting `MXFP4`, `FP8`, `GPTQ`, `AWQ`, or `AQLM`) and applies the corresponding configuration automatically.

3. **Runtime Quantization** – When `model_args.quantization_bit` is provided at runtime, the specified `quantization_method` (`BNB`, `HQQ`, or `EETQ`) determines which quantization library is instantiated.

## Configuring Quantization in Practice

To apply quantization in LlamaFactory, specify the method and bit precision through model arguments or training configuration files.

```python

# Runtime quantization with BNB 4-bit

from llamafactory.extras.constants import QuantizationMethod

model_args = {
    "quantization_bit": 4,
    "quantization_method": QuantizationMethod.BNB
}

# PTQ export with GPTQ

export_args = {
    "export_quantization_bit": 4,
    "quantization_method": QuantizationMethod.GPTQ
}

```

For pre-quantized model loading, LlamaFactory automatically detects the quantization method from the model's [`config.json`](https://github.com/hiyouga/LlamaFactory/blob/main/config.json) and applies the appropriate handler in `configure_quantization()`.

## Summary

- LlamaFactory supports nine quantization methods: **BNB**, **GPTQ**, **AWQ**, **AQLM**, **EETQ**, **HQQ**, **MXFP4**, **FP8**, and **QUANTO** (placeholder)
- Runtime methods (BNB, EETQ, HQQ) quantize models on-the-fly during loading
- PTQ methods (GPTQ, AWQ, AQLM, MXFP4, FP8) require export or pre-quantized checkpoints
- Method selection logic is centralized in `configure_quantization()` in [`src/llamafactory/model/model_utils/quantization.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/model/model_utils/quantization.py)
- The `QuantizationMethod` enum in [`src/llamafactory/extras/constants.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/extras/constants.py) (lines 135-147) defines all supported techniques

## Frequently Asked Questions

### What is the difference between BNB and GPTQ in LlamaFactory?

**BNB (BitsAndBytes)** performs quantization at runtime when loading the model, requiring no pre-processing and supporting 4-bit and 8-bit formats. **GPTQ** is primarily used for post-training quantization exports (2/3/4/8-bit) or loading pre-quantized GPTQ models, offering higher compression ratios but requiring the `optimum` library and export time processing.

### Can I use AWQ quantization for training in LlamaFactory?

**AWQ** in LlamaFactory is designed for inference and export scenarios using pre-quantized models or the `autoawq` library. While you can load AWQ-quantized models for fine-tuning with adapters, the quantization itself is applied post-training or via pre-quantized checkpoints, not during the training forward pass.

### Which LlamaFactory quantization method offers the best memory efficiency?

**AQLM** provides the highest compression with 2-bit quantization by default (configurable in [`quantization.py`](https://github.com/hiyouga/LlamaFactory/blob/main/quantization.py) lines 125-128), followed by **HQQ** which supports bit widths as low as 1-bit. For runtime quantization without pre-processing, **BNB** 4-bit (NF4) offers excellent memory reduction with minimal accuracy loss.

### How do I check if my model is using the correct quantization method?

LlamaFactory validates quantization configuration in `configure_quantization()` (lines 82-95). Check the model's `quantization_config` in the generated config files, or inspect the logs during initialization where the library reports which `QuantizationMethod` enum value is active and the corresponding config class being instantiated (e.g., `BitsAndBytesConfig`, `EetqConfig`, or `HqqConfig`).