INT8, GPTQ, AWQ, and GGUF Quantization Methods in Phase 10 of AI Engineering From Scratch

Phase 10 of the ai-engineering-from-scratch curriculum details four production quantization techniques—INT8, GPTQ, AWQ, and GGUF—that enable deploying large language models with 2× to 4× memory reduction through integer scaling, Hessian-guided layer optimization, activation-aware weight protection, and CPU-optimized mixed-precision formats.

The "Quantization" lesson (Lesson 11) in Phase 10 of the rohitg00/ai-engineering-from-scratch repository provides concrete implementations of model compression strategies widely used in inference engines like vLLM, TensorRT-LLM, and llama.cpp. According to the curriculum documentation in phases/10-llms-from-scratch/11-quantization/docs/en.md, these methods balance memory efficiency against accuracy loss through distinct mathematical approaches ranging from symmetric integer quantization to salient-weight preservation algorithms.

INT8 Quantization: Uniform 8-Bit Compression

INT8 quantization converts 32-bit floating-point weights into 8-bit integers using a symmetric scale factor, achieving approximately 2× memory reduction with less than 0.5% quality degradation. As documented in phases/10-llms-from-scratch/11-quantization/docs/en.md#L53-L55, this method utilizes 256 uniform steps (values -127 to 127) with a per-tensor scaling factor derived from the maximum absolute value.

This approach is optimized for GPU inference via frameworks like vLLM and TensorRT-LLM, where the reduced memory bandwidth directly translates to higher throughput. The quantization process involves calculating a scale factor, clipping values to the INT8 range, and converting back to floating-point during dequantization.

NumPy Implementation

The lesson provides a pure NumPy implementation demonstrating symmetric quantization:

import numpy as np

def quantize_int8(tensor):
    qmin, qmax = -127, 127
    scale = np.max(np.abs(tensor)) / qmax
    quantized = np.clip(np.round(tensor / scale), qmin, qmax).astype(np.int8)
    return quantized, scale

def dequantize_int8(q, scale):
    return q.astype(np.float32) * scale

GPTQ: Hessian-Guided INT4 Quantization

GPTQ (General-purpose Post-Training Quantization) enables aggressive 4-bit (INT4) compression by using a small calibration set to compute Hessian-guided per-layer scaling factors. As detailed in phases/10-llms-from-scratch/11-quantization/docs/en.md#L59-L61, this post-training quantization (PTQ) technique is essential when deploying 70B parameter models on single 48GB GPUs, achieving roughly 4× memory savings compared to FP16.

GPTQ analyzes the layer-wise Hessian matrix to determine optimal scaling that minimizes the error introduced by quantization, processing weights in arbitrary order while accounting for quantization errors propagated to subsequent weights.

AutoGPTQ Implementation

The curriculum demonstrates GPTQ using the auto-gptq library with a 4-bit configuration and group size of 128:

# pip install auto-gptq transformers

from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
from transformers import AutoTokenizer

model_id = "meta-llama/Llama-3.1-8B"
quantize_cfg = BaseQuantizeConfig(bits=4, group_size=128, desc_act=False)

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoGPTQForCausalLM.from_pretrained(model_id, quantize_cfg)

# 128-example calibration set (omitted for brevity)

model.quantize(calibration)
model.save_quantized("llama-8b-gptq-int4")

AWQ: Activation-Aware Weight Protection

AWQ (Activation-Aware Weight Quantization) improves upon GPTQ by identifying and protecting the approximately 1% most salient weights—those that multiply large activations—scaling them up before quantization to preserve model quality. According to phases/10-llms-from-scratch/11-quantization/docs/en.md#L61-L62, this method achieves comparable INT4 compression to GPTQ while being 1.5–2× faster to apply during the quantization process.

The activation-aware approach recognizes that not all weights contribute equally to model output; by preserving high-impact weights in higher precision while quantizing the remainder, AWQ maintains accuracy without requiring the computational overhead of Hessian matrix calculations.

AutoAWQ Implementation

The lesson provides an autoawq implementation demonstrating zero-point quantization with 128-size groups:

# pip install autoawq

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_id = "meta-llama/Llama-3.1-8B"
model = AutoAWQForCausalLM.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)

model.quantize(tokenizer, quant_config={"zero_point": True,
                                        "q_group_size": 128,
                                        "w_bit": 4})
model.save_quantized("llama-8b-awq-int4")

GGUF: Mixed-Precision CPU Inference

GGUF (GPT-Generated Unified Format) is the native format for llama.cpp, packing mixed-precision weights (such as Q4_K_M and Q5_K_M) into a single file optimized for CPU or Apple Silicon inference. As described in phases/10-llms-from-scratch/11-quantization/docs/en.md#L63-L65, this format is specifically designed for edge devices and laptops where GPU acceleration is unavailable.

Unlike uniform quantization methods, GGUF employs different quantization strategies for different tensor types (attention queries vs. feed-forward networks) to optimize the quality-to-size ratio for CPU inference engines.

llama.cpp Conversion and Serving

The curriculum demonstrates converting HuggingFace models to GGUF format and serving them:

# pip install llama-cpp-python

python convert_hf_to_gguf.py meta-llama/Llama-3.1-8B \
    --outtype q4_k_m \
    --outfile llama-8b-q4km.gguf

# Serve with the llama.cpp server

llama-server -m llama-8b-q4km.gguf -c 4096 -ngl 99

Selecting the Right Quantization Method

The decision framework presented in phases/10-llms-from-scratch/11-quantization/outputs/skill-quantization.md recommends selecting quantization methods based on hardware constraints and quality requirements:

• INT8: Use for GPU deployments requiring maximum speed with minimal accuracy loss
• GPTQ: Use for aggressive GPU memory reduction when calibration data is available
• AWQ: Use for faster quantization workflows while maintaining GPTQ-level quality
• GGUF: Use for CPU-only inference environments and edge deployment

Summary

• INT8 quantization provides a 2× memory reduction through symmetric 8-bit integer representation with <0.5% quality loss, ideal for high-throughput GPU inference
• GPTQ enables 4-bit compression via Hessian-guided optimization, making 70B models viable on consumer GPUs through post-training calibration
• AWQ accelerates the quantization process by protecting salient weights based on activation magnitudes, offering comparable compression to GPTQ with faster processing times
• GGUF delivers mixed-precision CPU-optimized formats through llama.cpp, supporting edge deployment on laptops and Apple Silicon devices
• All four methods are implemented with runnable code examples in phases/10-llms-from-scratch/11-quantization/code/quantization_demo.py and documented in the Phase 10 quantization lesson

Frequently Asked Questions

What is the difference between GPTQ and AWQ quantization?

GPTQ uses a Hessian matrix derived from calibration data to guide per-layer quantization scales, while AWQ identifies the top 1% of weights by activation magnitude and scales them up before quantization to preserve accuracy. According to the Phase 10 curriculum in phases/10-llms-from-scratch/11-quantization/docs/en.md, AWQ achieves similar compression ratios to GPTQ but processes approximately 1.5–2× faster because it avoids computationally expensive Hessian calculations.

When should I use GGUF instead of GPTQ or AWQ?

Use GGUF when deploying models on CPU-only environments or edge devices such as laptops and Apple Silicon Macs, as the format is specifically optimized for llama.cpp inference engines. GPTQ and AWQ are designed for GPU acceleration and will not run efficiently on CPU-only systems, whereas GGUF formats like Q4_K_M provide mixed-precision compression tailored for CPU memory bandwidth constraints.

Does INT8 quantization require retraining the model?

No, INT8 quantization is a post-training technique that applies symmetric scaling to pre-trained weights without requiring gradient updates or fine-tuning. As implemented in phases/10-llms-from-scratch/11-quantization/code/quantization_demo.py, the method calculates a simple per-tensor scale factor based on maximum absolute values, enabling immediate deployment with less than 0.5% accuracy degradation.

Which quantization method offers the best memory reduction?

GPTQ and AWQ both achieve 4-bit (INT4) compression, providing approximately 4× memory reduction compared to FP16 models, making them optimal for maximizing memory efficiency. While GGUF also supports 4-bit variants like Q4_K_M, the uniform INT8 format offers only 2× reduction but maintains higher precision for applications where accuracy is critical.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}