# Quantization Methods for LLM Optimization: INT8, GPTQ, AWQ, and GGUF Explained

> Explore INT8, GPTQ, AWQ, and GGUF quantization for LLM optimization. Reduce memory 2-4x with minimal inference quality loss using production-ready implementations. Optimize your AI models today.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: deep-dive
- Published: 2026-07-31

---

**The ai-engineering-from-scratch repository delivers production-ready implementations of INT8, GPTQ, AWQ, and GGUF quantization methods, enabling 2–4× memory reduction for large language models while preserving inference quality across diverse hardware platforms.**

The `rohitg00/ai-engineering-from-scratch` repository covers quantization methods for LLM optimization in Phase 10, Lesson 11, providing architecture-level explanations and executable code that transforms floating-point models into compressed integer formats. This module, documented in [`phases/10-llms-from-scratch/11-quantization/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/11-quantization/docs/en.md), includes parallel Python and Rust implementations demonstrating per-tensor, per-channel, and simulated 4-bit quantization pipelines.

## The Four Quantization Pathways

The repository treats quantization as a spectrum of trade-offs between memory footprint, computational speed, and model fidelity. Each method targets specific deployment constraints, from high-throughput GPU serving to CPU-powered edge devices.

### INT8 (Symmetric 8-bit Weight-Only)

**INT8** quantization represents weights as uniform 8-bit integers using a single scale factor per tensor or channel, typically achieving a **2× memory reduction** compared to FP16. This method relies on simple symmetric scaling where the scale equals the maximum absolute value divided by 127.

According to the source documentation in [`phases/10-llms-from-scratch/11-quantization/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/11-quantization/docs/en.md), INT8 serves as the baseline for general-purpose GPU serving where near-perfect model quality is required and only modest compression is needed. The reference implementation in [`phases/10-llms-from-scratch/11-quantization/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/11-quantization/code/main.py) demonstrates both per-tensor and per-channel variants without requiring calibration data, though quality may degrade for outlier weights without fine-tuning.

### GPTQ (Hessian-Guided Post-Training Quantization)

**GPTQ** enables aggressive **4-bit (INT4)** compression using a post-training, layer-wise approach that minimizes error through Hessian-based optimization. The algorithm processes weights sequentially, using a calibration dataset of approximately **128 examples** to compute second-order statistics and allocate rounding error to less-important weights.

As documented in the quantization lesson, GPTQ is the preferred method when serving very large models (≥70B parameters) with **vLLM and multi-LoRA support**, delivering a **4× memory reduction** while maintaining adapter compatibility. The implementation relies on the `auto_gptq` library, with configuration parameters including `bits=4`, `group_size=128`, and `desc_act=False` controlling the quantization granularity.

### AWQ (Activation-Aware Weight Quantization)

**AWQ** protects model quality by identifying the approximately **1% of weights** that interact with large activations (salient weights). Before applying INT4 rounding, AWQ scales these critical weights up and correspondingly scales down the associated activations, preventing accuracy degradation in sensitive channels.

The repository notes that AWQ converts models approximately **1.5–2× faster** than GPTQ and is optimal for GPU serving scenarios where multi-LoRA support is unnecessary. Like GPTQ, AWQ requires a small calibration set (typically 128–2,000 samples) to detect activation magnitudes, as implemented in the `AutoAWQForCausalLM` class workflow.

### GGUF (Mixed-Precision Container Format)

**GGUF** is the container format utilized by [`llama.cpp`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/llama.cpp) and `llama-cpp-python`, supporting **mixed-precision** quantization where early and late layers retain higher precision (e.g., Q5_K_M) while internal layers run at Q4_K_M. This selective approach balances memory reduction with output quality on resource-constrained devices.

Documented in [`phases/10-llms-from-scratch/11-quantization/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/11-quantization/docs/en.md), GGUF is explicitly optimized for **CPU and Apple Silicon inference**, serving as the default format for the Ollama ecosystem. Unlike GPU-focused methods, GGUF leverages hand-tuned integer kernels for ARM and x86 architectures, making it the standard for laptop and edge deployment where CUDA acceleration is unavailable.

## Calibration Requirements and Hardware Alignment

The choice of quantization method depends on calibration data availability and target hardware, considerations detailed in [`phases/17-infrastructure-and-production/09-production-quantization/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/17-infrastructure-and-production/09-production-quantization/docs/en.md).

### Calibration Dependencies

- **GPTQ and AWQ** require calibration datasets (128–2,000 samples) to compute Hessian matrices or activation statistics.
- **INT8** weight-only quantization can execute without calibration, though per-channel scaling may need statistical analysis of weight distributions.
- **GGUF** conversion typically uses pre-calibrated templates (Q4_K_M, Q5_K_M) that require no user-provided dataset.

### Platform-Specific Recommendations

- **GPU Servers (vLLM, TensorRT-LLM):** Prefer **GPTQ** for multi-LoRA pipelines or **AWQ** for maximum throughput without adapter support.
- **CPU / Apple Silicon:** **GGUF** is the exclusive choice, optimized through [`llama.cpp`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/llama.cpp) kernels for non-CUDA inference.
- **Hybrid Pipelines:** INT8 weights with FP16 outliers (LLM.int8()) represent an alternative hybrid, though the repository focuses on the four standalone methods above.

## Implementation Examples

The repository provides runnable code paths in [`phases/10-llms-from-scratch/11-quantization/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/11-quantization/code/main.py) (Python) and [`phases/10-llms-from-scratch/11-quantization/code/main.rs`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/11-quantization/code/main.rs) (Rust) demonstrating each quantization technique.

### INT8 Symmetric Quantization

```python
import numpy as np

def quantize_int8(tensor):
    """Per-tensor symmetric INT8 quantization."""
    qmin, qmax = -127, 127
    scale = np.max(np.abs(tensor)) / qmax
    q = np.clip(np.round(tensor / scale), qmin, qmax).astype(np.int8)
    return q, scale

```

### GPTQ 4-bit Conversion

```python
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig

model_id = "meta-llama/Llama-3.1-8B"
cfg = BaseQuantizeConfig(bits=4, group_size=128, desc_act=False)

model = AutoGPTQForCausalLM.from_pretrained(model_id, quantize_config=cfg)
model.quantize(calibration_dataset)  # 128 examples required

model.save_quantized("llama-8b-gptq-int4")

```

### AWQ Activation-Aware Quantization

```python
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_id = "meta-llama/Llama-3.1-8B"
model = AutoAWQForCausalLM.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)

quant_config = {"zero_point": True, "q_group_size": 128, "w_bit": 4}
model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized("llama-8b-awq-int4")

```

### GGUF Conversion for CPU Inference

```bash

# Install llama.cpp conversion tools

pip install llama-cpp-python

# Convert HF checkpoint to Q4_K_M mixed-precision GGUF

python convert_hf_to_gguf.py meta-llama/Llama-3.1-8B \
    --outtype q4_k_m \
    --outfile llama-8b-q4km.gguf

# Serve with built-in server

llama-server -m llama-8b-q4km.gguf -c 4096 -ngl 99

```

## Summary

- **INT8** provides the fastest calibration-free path to 2× memory reduction for GPU inference.
- **GPTQ** achieves maximum 4× compression using Hessian calibration, essential for multi-LoRA serving with vLLM.
- **AWQ** delivers superior inference speed with 4-bit precision by protecting activation-sensitive weights, ideal for single-model GPU deployment.
- **GGUF** enables mixed-precision CPU and edge deployment through the [`llama.cpp`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/llama.cpp) ecosystem, supporting the Ollama framework.

## Frequently Asked Questions

### What is the difference between GPTQ and AWQ?

**GPTQ** utilizes Hessian-based calibration to minimize quantization error across all weights equally, making it compatible with vLLM's multi-LoRA serving. **AWQ** selectively preserves the 1% of weights that interact with large activations, resulting in 1.5–2× faster conversion speeds but no support for multiple LoRA adapters simultaneously.

### Does INT8 quantization require a calibration dataset?

INT8 weight-only quantization can be performed without calibration using simple per-tensor or per-channel scaling, as shown in the reference [`main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/main.py). However, for optimal quality on models with significant outlier weights, a small calibration set helps determine optimal scaling factors.

### Can GGUF models run on GPU?

While GGUF is optimized for **CPU and Apple Silicon** inference via [`llama.cpp`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/llama.cpp), the format supports GPU offloading through the `-ngl` (n-gpu-layers) parameter in `llama-server`. However, for dedicated GPU serving, GPTQ or AWQ provide superior throughput through native CUDA kernels.

### Which quantization method is best for edge deployment?

**GGUF** is the recommended method for edge and laptop deployment, particularly when using the [`llama.cpp`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/llama.cpp) runtime or Ollama ecosystem. Its mixed-precision support (Q4_K_M, Q5_K_M) and hand-tuned CPU kernels allow efficient inference on devices without CUDA-capable GPUs.