Quantization Methods for LLM Optimization: INT8, GPTQ, AWQ, and GGUF Explained

The ai-engineering-from-scratch repository delivers production-ready implementations of INT8, GPTQ, AWQ, and GGUF quantization methods, enabling 2–4× memory reduction for large language models while preserving inference quality across diverse hardware platforms.

The rohitg00/ai-engineering-from-scratch repository covers quantization methods for LLM optimization in Phase 10, Lesson 11, providing architecture-level explanations and executable code that transforms floating-point models into compressed integer formats. This module, documented in phases/10-llms-from-scratch/11-quantization/docs/en.md, includes parallel Python and Rust implementations demonstrating per-tensor, per-channel, and simulated 4-bit quantization pipelines.

The Four Quantization Pathways

The repository treats quantization as a spectrum of trade-offs between memory footprint, computational speed, and model fidelity. Each method targets specific deployment constraints, from high-throughput GPU serving to CPU-powered edge devices.

INT8 (Symmetric 8-bit Weight-Only)

INT8 quantization represents weights as uniform 8-bit integers using a single scale factor per tensor or channel, typically achieving a 2× memory reduction compared to FP16. This method relies on simple symmetric scaling where the scale equals the maximum absolute value divided by 127.

According to the source documentation in phases/10-llms-from-scratch/11-quantization/docs/en.md, INT8 serves as the baseline for general-purpose GPU serving where near-perfect model quality is required and only modest compression is needed. The reference implementation in phases/10-llms-from-scratch/11-quantization/code/main.py demonstrates both per-tensor and per-channel variants without requiring calibration data, though quality may degrade for outlier weights without fine-tuning.

GPTQ (Hessian-Guided Post-Training Quantization)

GPTQ enables aggressive 4-bit (INT4) compression using a post-training, layer-wise approach that minimizes error through Hessian-based optimization. The algorithm processes weights sequentially, using a calibration dataset of approximately 128 examples to compute second-order statistics and allocate rounding error to less-important weights.

As documented in the quantization lesson, GPTQ is the preferred method when serving very large models (≥70B parameters) with vLLM and multi-LoRA support, delivering a 4× memory reduction while maintaining adapter compatibility. The implementation relies on the auto_gptq library, with configuration parameters including bits=4, group_size=128, and desc_act=False controlling the quantization granularity.

AWQ (Activation-Aware Weight Quantization)

AWQ protects model quality by identifying the approximately 1% of weights that interact with large activations (salient weights). Before applying INT4 rounding, AWQ scales these critical weights up and correspondingly scales down the associated activations, preventing accuracy degradation in sensitive channels.

The repository notes that AWQ converts models approximately 1.5–2× faster than GPTQ and is optimal for GPU serving scenarios where multi-LoRA support is unnecessary. Like GPTQ, AWQ requires a small calibration set (typically 128–2,000 samples) to detect activation magnitudes, as implemented in the AutoAWQForCausalLM class workflow.

GGUF (Mixed-Precision Container Format)

GGUF is the container format utilized by llama.cpp and llama-cpp-python, supporting mixed-precision quantization where early and late layers retain higher precision (e.g., Q5_K_M) while internal layers run at Q4_K_M. This selective approach balances memory reduction with output quality on resource-constrained devices.

Documented in phases/10-llms-from-scratch/11-quantization/docs/en.md, GGUF is explicitly optimized for CPU and Apple Silicon inference, serving as the default format for the Ollama ecosystem. Unlike GPU-focused methods, GGUF leverages hand-tuned integer kernels for ARM and x86 architectures, making it the standard for laptop and edge deployment where CUDA acceleration is unavailable.

Calibration Requirements and Hardware Alignment

The choice of quantization method depends on calibration data availability and target hardware, considerations detailed in phases/17-infrastructure-and-production/09-production-quantization/docs/en.md.

Calibration Dependencies

• GPTQ and AWQ require calibration datasets (128–2,000 samples) to compute Hessian matrices or activation statistics.
• INT8 weight-only quantization can execute without calibration, though per-channel scaling may need statistical analysis of weight distributions.
• GGUF conversion typically uses pre-calibrated templates (Q4_K_M, Q5_K_M) that require no user-provided dataset.

Platform-Specific Recommendations

• GPU Servers (vLLM, TensorRT-LLM): Prefer GPTQ for multi-LoRA pipelines or AWQ for maximum throughput without adapter support.
• CPU / Apple Silicon: GGUF is the exclusive choice, optimized through llama.cpp kernels for non-CUDA inference.
• Hybrid Pipelines: INT8 weights with FP16 outliers (LLM.int8()) represent an alternative hybrid, though the repository focuses on the four standalone methods above.

Implementation Examples

The repository provides runnable code paths in phases/10-llms-from-scratch/11-quantization/code/main.py (Python) and phases/10-llms-from-scratch/11-quantization/code/main.rs (Rust) demonstrating each quantization technique.

INT8 Symmetric Quantization

import numpy as np

def quantize_int8(tensor):
    """Per-tensor symmetric INT8 quantization."""
    qmin, qmax = -127, 127
    scale = np.max(np.abs(tensor)) / qmax
    q = np.clip(np.round(tensor / scale), qmin, qmax).astype(np.int8)
    return q, scale

GPTQ 4-bit Conversion

from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig

model_id = "meta-llama/Llama-3.1-8B"
cfg = BaseQuantizeConfig(bits=4, group_size=128, desc_act=False)

model = AutoGPTQForCausalLM.from_pretrained(model_id, quantize_config=cfg)
model.quantize(calibration_dataset)  # 128 examples required

model.save_quantized("llama-8b-gptq-int4")

AWQ Activation-Aware Quantization

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_id = "meta-llama/Llama-3.1-8B"
model = AutoAWQForCausalLM.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)

quant_config = {"zero_point": True, "q_group_size": 128, "w_bit": 4}
model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized("llama-8b-awq-int4")

GGUF Conversion for CPU Inference

# Install llama.cpp conversion tools

pip install llama-cpp-python

# Convert HF checkpoint to Q4_K_M mixed-precision GGUF

python convert_hf_to_gguf.py meta-llama/Llama-3.1-8B \
    --outtype q4_k_m \
    --outfile llama-8b-q4km.gguf

# Serve with built-in server

llama-server -m llama-8b-q4km.gguf -c 4096 -ngl 99

Summary

• INT8 provides the fastest calibration-free path to 2× memory reduction for GPU inference.
• GPTQ achieves maximum 4× compression using Hessian calibration, essential for multi-LoRA serving with vLLM.
• AWQ delivers superior inference speed with 4-bit precision by protecting activation-sensitive weights, ideal for single-model GPU deployment.
• GGUF enables mixed-precision CPU and edge deployment through the llama.cpp ecosystem, supporting the Ollama framework.

Frequently Asked Questions

What is the difference between GPTQ and AWQ?

GPTQ utilizes Hessian-based calibration to minimize quantization error across all weights equally, making it compatible with vLLM's multi-LoRA serving. AWQ selectively preserves the 1% of weights that interact with large activations, resulting in 1.5–2× faster conversion speeds but no support for multiple LoRA adapters simultaneously.

Does INT8 quantization require a calibration dataset?

INT8 weight-only quantization can be performed without calibration using simple per-tensor or per-channel scaling, as shown in the reference main.py. However, for optimal quality on models with significant outlier weights, a small calibration set helps determine optimal scaling factors.

Can GGUF models run on GPU?

While GGUF is optimized for CPU and Apple Silicon inference via llama.cpp, the format supports GPU offloading through the -ngl (n-gpu-layers) parameter in llama-server. However, for dedicated GPU serving, GPTQ or AWQ provide superior throughput through native CUDA kernels.

Which quantization method is best for edge deployment?

GGUF is the recommended method for edge and laptop deployment, particularly when using the llama.cpp runtime or Ollama ecosystem. Its mixed-precision support (Q4_K_M, Q5_K_M) and hand-tuned CPU kernels allow efficient inference on devices without CUDA-capable GPUs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}