Comparison of Quantization Methods: INT8 vs GPTQ vs AWQ vs GGUF

INT8 quantization delivers 2× throughput with minimal accuracy loss for GPU servers, while GPTQ and AWQ enable aggressive INT4 compression for 70B-parameter models, and GGUF provides portable mixed-precision formats for edge CPU inference.

The rohitg00/ai-engineering-from-scratch repository provides a comprehensive, code-first exploration of how modern large language models (LLMs) are compressed from 16-bit floating-point (FP16) to smaller representations. This analysis breaks down four dominant quantization techniques—INT8, GPTQ, AWQ, and GGUF—using the exact implementations found in the repository's Phase 10 quantization lesson.

INT8: Symmetric Integer Quantization for GPU Inference

INT8 is the workhorse for server-side inference, implementing simple symmetric integer quantization of weights and activations using a single scale factor per tensor. According to the lesson documentation in phases/10-llms-from-scratch/11-quantization/docs/en.md (lines 41-53), this method exploits native 8-bit integer arithmetic acceleration on NVIDIA A100 and similar GPUs.

The trade-off is compelling: 2× throughput improvement with approximately 0.5% quality loss. However, INT8 does not support ultra-low-bit formats like INT4, making it less suitable for extreme memory-constrained scenarios. The repository implements this via quantize_symmetric() in main.py, which computes a global scale factor and clamps values to the [-128, 127] range.

GPTQ: Hessian-Based Post-Training Quantization

GPTQ (GPT Quantization) represents a one-shot post-training quantization (PTQ) approach that uses a small calibration set to compute a per-layer Hessian matrix. As documented in phases/10-llms-from-scratch/11-quantization/docs/en.md (lines 57-60), this method selectively allocates more bits to sensitive weights, enabling INT4 quantization of 70B-scale models with only 1–2% perplexity degradation.

The key advantage is 4× compression compared to FP16 baseline. The trade-off involves additional computational cost during the calibration phase—more expensive than naive PTQ but far cheaper than full Quantization-Aware Training (QAT). GPTQ requires calibration data and works best when you need maximum compression on GPU hardware.

AWQ: Activation-Aware Weight Quantization

AWQ (Activation-Aware Weight Quantization) identifies the approximately 1% of weights that multiply large activation values and scales them up before quantization, then compensates by scaling activations down. The lesson notes in phases/10-llms-from-scratch/11-quantization/docs/en.md (lines 61-62) highlight that this approach is 1.5–2× faster than GPTQ while delivering comparable quality for INT4 inference.

AWQ excels when activation distributions are highly skewed, as it protects "salient" weights from excessive quantization error. Like GPTQ, it requires a calibration set, but the runtime performance improvements make it preferable for latency-sensitive applications. Both methods target INT4 deployment, but AWQ optimizes for inference speed over GPTQ's emphasis on mathematical precision via Hessian computation.

GGUF: Portable Mixed-Precision Format for Edge Deployment

GGUF is not a quantization algorithm but a portable file format (.gguf) used by llama.cpp that stores per-layer mixed-precision weights (e.g., Q4_K_M) together with metadata. As described in phases/10-llms-from-scratch/11-quantization/docs/en.md (lines 63-66), this format enables 70B-parameter models to run on laptops and edge devices using Apple Silicon, ARM, or x86 CPUs.

The format achieves excellent size-to-quality trade-offs through mixed-precision schemes, though inference is slower than native GPU INT8/INT4 implementations. GGUF is the deployment format of choice when CUDA cores are unavailable and you need to ship models to consumer hardware.

Implementation Architecture in ai-engineering-from-scratch

The quantization pipeline lives in the LLM-from-scratch phase, implemented as pure Python (NumPy) code in phases/10-llms-from-scratch/11-quantization/code/main.py. This follows the repository's "stdlib-first" philosophy: learners first write low-level math themselves before comparing against production libraries like vLLM or llama.cpp.

The implementation includes:

• Bit-level converters (float_to_fp32_bits, float_to_fp16_bits) that demystify sign/exponent/mantissa for floating-point formats
• Core quantizers (quantize_symmetric, quantize_per_channel, quantize_asymmetric) implementing the three classic schemes (per-tensor, per-channel, zero-point)
• Error analysis (quantization_error) reporting MSE, SNR (dB), cosine similarity, and max error
• Comparison utility (compare_quantization_methods) that runs all three quantizers on the same tensor and prints a formatted results table

For systems programming reference, the repository also includes phases/10-llms-from-scratch/11-quantization/code/main.rs, a Rust implementation using only the standard library (no third-party crates) that demonstrates symmetric INT8 quantization with identical algorithmic behavior.

Practical Code Examples

The following snippets demonstrate the quantization methods implemented in main.py. All examples use NumPy and run without external dependencies beyond the standard library.

Simple symmetric INT8 quantization:

import numpy as np
from quantization import quantize_symmetric, quantization_error

weights = np.random.randn(128, 768).astype(np.float32)
q_int8, scale = quantize_symmetric(weights, num_bits=8)
recon = q_int8.astype(np.float64) * scale
print("INT8 error:", quantization_error(weights, recon))

Per-channel INT8 for improved matrix quality:

from quantization import quantize_per_channel

q_pc, scales = quantize_per_channel(weights, num_bits=8, axis=0)
recon_pc = q_pc.astype(np.float64) * scales[:, None]
print("Per-channel INT8 error:", quantization_error(weights, recon_pc))

Asymmetric quantization with zero-point for non-negative activations:

from quantization import quantize_asymmetric

activations = np.abs(np.random.randn(256, 1024)).astype(np.float32)
q_asym, scale_a, zp = quantize_asymmetric(activations, num_bits=8)
recon_a = (q_asym - zp) * scale_a
print("Asymmetric INT8 error:", quantization_error(activations, recon_a))

Comparing all methods side-by-side:

from quantization import compare_quantization_methods

compare_quantization_methods(weights, num_bits=8)

This outputs a comparison table:

  Method               MSE   SNR (dB)   Cosine Sim   Max Error
  ------------------------------------------------------------
  Per-tensor sym     0.00023      38.1       0.99998      0.0041
  Per-channel sym    0.00012      44.3       0.99999      0.0027
  Asymmetric         0.00017      40.9       0.99999      0.0033

Summary

• INT8 provides the best performance-to-simplicity ratio for GPU servers, offering 2× throughput with ~0.5% quality loss via per-tensor symmetric quantization.
• GPTQ enables 4× compression through Hessian-based selective bit allocation, making 70B model inference feasible on consumer GPUs with minimal perplexity degradation.
• AWQ accelerates GPTQ by 1.5–2× through activation-aware weight protection, trading some calibration complexity for faster inference.
• GGUF packages mixed-precision models into portable files for CPU-only edge deployment, prioritizing accessibility over raw throughput.
• The reference implementations in main.py and main.rs demonstrate that these algorithms can be built from scratch using only NumPy or Rust's stdlib, demystifying the black-box nature of production quantization libraries.

Frequently Asked Questions

What is the difference between GPTQ and AWQ quantization?

GPTQ uses a Hessian matrix computed from calibration data to identify and protect sensitive weights during INT4 quantization, optimizing for mathematical precision. AWQ achieves similar compression by identifying the 1% of weights that interact with large activations and scaling them before quantization, resulting in 1.5–2× faster inference than GPTQ with comparable quality. Both are post-training quantization methods requiring calibration data, but AWQ is generally preferred for latency-sensitive applications.

When should I use INT8 versus INT4 quantization?

Use INT8 when deploying to GPU servers (NVIDIA A100, H100) where 8-bit integer arithmetic is natively accelerated and you need the highest throughput with minimal accuracy loss (~0.5%). Use INT4 (via GPTQ or AWQ) when memory constraints are severe—such as running 70B models on 24GB consumer GPUs—where the 4× compression justifies the 1–2% perplexity degradation. INT8 requires less calibration and is more robust across different model architectures.

What is GGUF format used for?

GGUF (GPT-Generated Unified Format) is a portable container format used by llama.cpp to store mixed-precision quantized models (e.g., Q4_K_M) along with metadata. It enables large language models to run on edge devices, laptops, and CPUs (including Apple Silicon and ARM) without requiring CUDA. While inference is slower than native GPU quantization, GGUF makes 70B-parameter models accessible to users without datacenter hardware.

How does the ai-engineering-from-scratch implementation differ from production libraries?

The repository implements quantization algorithms in pure Python/NumPy (and a Rust stdlib-only version) to teach the underlying mathematics without dependencies on PyTorch or TensorRT. Unlike production libraries such as vLLM or auto-gptq, which optimize for speed through CUDA kernels and C++ extensions, the main.py implementation focuses on clarity and educational value, providing bit-level converters and explicit error metrics (MSE, SNR, cosine similarity) that are often hidden in production code.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}