# Comparison of Quantization Methods: INT8 vs GPTQ vs AWQ vs GGUF

> Compare INT8, GPTQ, AWQ, and GGUF quantization methods. Discover optimal techniques for GPU throughput, aggressive INT4 compression, and portable CPU inference for large language models.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: deep-dive
- Published: 2026-06-24

---

**INT8 quantization delivers 2× throughput with minimal accuracy loss for GPU servers, while GPTQ and AWQ enable aggressive INT4 compression for 70B-parameter models, and GGUF provides portable mixed-precision formats for edge CPU inference.**

The **rohitg00/ai-engineering-from-scratch** repository provides a comprehensive, code-first exploration of how modern large language models (LLMs) are compressed from 16-bit floating-point (FP16) to smaller representations. This analysis breaks down four dominant quantization techniques—INT8, GPTQ, AWQ, and GGUF—using the exact implementations found in the repository's Phase 10 quantization lesson.

## INT8: Symmetric Integer Quantization for GPU Inference

**INT8** is the workhorse for server-side inference, implementing simple symmetric integer quantization of weights and activations using a single scale factor per tensor. According to the lesson documentation in [`phases/10-llms-from-scratch/11-quantization/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/11-quantization/docs/en.md) (lines 41-53), this method exploits native 8-bit integer arithmetic acceleration on NVIDIA A100 and similar GPUs.

The trade-off is compelling: **2× throughput** improvement with approximately **0.5% quality loss**. However, INT8 does not support ultra-low-bit formats like INT4, making it less suitable for extreme memory-constrained scenarios. The repository implements this via `quantize_symmetric()` in [`main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/main.py), which computes a global scale factor and clamps values to the `[-128, 127]` range.

## GPTQ: Hessian-Based Post-Training Quantization

**GPTQ** (GPT Quantization) represents a one-shot post-training quantization (PTQ) approach that uses a small calibration set to compute a per-layer Hessian matrix. As documented in [`phases/10-llms-from-scratch/11-quantization/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/11-quantization/docs/en.md) (lines 57-60), this method selectively allocates more bits to sensitive weights, enabling **INT4 quantization** of 70B-scale models with only **1–2% perplexity degradation**.

The key advantage is **4× compression** compared to FP16 baseline. The trade-off involves additional computational cost during the calibration phase—more expensive than naive PTQ but far cheaper than full Quantization-Aware Training (QAT). GPTQ requires calibration data and works best when you need maximum compression on GPU hardware.

## AWQ: Activation-Aware Weight Quantization

**AWQ** (Activation-Aware Weight Quantization) identifies the approximately **1% of weights** that multiply large activation values and scales them up before quantization, then compensates by scaling activations down. The lesson notes in [`phases/10-llms-from-scratch/11-quantization/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/11-quantization/docs/en.md) (lines 61-62) highlight that this approach is **1.5–2× faster** than GPTQ while delivering comparable quality for INT4 inference.

AWQ excels when activation distributions are highly skewed, as it protects "salient" weights from excessive quantization error. Like GPTQ, it requires a calibration set, but the runtime performance improvements make it preferable for latency-sensitive applications. Both methods target INT4 deployment, but AWQ optimizes for inference speed over GPTQ's emphasis on mathematical precision via Hessian computation.

## GGUF: Portable Mixed-Precision Format for Edge Deployment

**GGUF** is not a quantization algorithm but a portable file format (`.gguf`) used by **llama.cpp** that stores per-layer mixed-precision weights (e.g., `Q4_K_M`) together with metadata. As described in [`phases/10-llms-from-scratch/11-quantization/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/11-quantization/docs/en.md) (lines 63-66), this format enables 70B-parameter models to run on laptops and edge devices using Apple Silicon, ARM, or x86 CPUs.

The format achieves excellent size-to-quality trade-offs through mixed-precision schemes, though inference is **slower than native GPU INT8/INT4** implementations. GGUF is the deployment format of choice when CUDA cores are unavailable and you need to ship models to consumer hardware.

## Implementation Architecture in ai-engineering-from-scratch

The quantization pipeline lives in the **LLM-from-scratch** phase, implemented as **pure Python (NumPy)** code in [`phases/10-llms-from-scratch/11-quantization/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/11-quantization/code/main.py). This follows the repository's "stdlib-first" philosophy: learners first write low-level math themselves before comparing against production libraries like `vLLM` or [`llama.cpp`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/llama.cpp).

The implementation includes:

- **Bit-level converters** (`float_to_fp32_bits`, `float_to_fp16_bits`) that demystify sign/exponent/mantissa for floating-point formats
- **Core quantizers** (`quantize_symmetric`, `quantize_per_channel`, `quantize_asymmetric`) implementing the three classic schemes (per-tensor, per-channel, zero-point)
- **Error analysis** (`quantization_error`) reporting MSE, SNR (dB), cosine similarity, and max error
- **Comparison utility** (`compare_quantization_methods`) that runs all three quantizers on the same tensor and prints a formatted results table

For systems programming reference, the repository also includes [`phases/10-llms-from-scratch/11-quantization/code/main.rs`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/11-quantization/code/main.rs), a Rust implementation using only the standard library (no third-party crates) that demonstrates symmetric INT8 quantization with identical algorithmic behavior.

## Practical Code Examples

The following snippets demonstrate the quantization methods implemented in [`main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/main.py). All examples use NumPy and run without external dependencies beyond the standard library.

Simple symmetric INT8 quantization:

```python
import numpy as np
from quantization import quantize_symmetric, quantization_error

weights = np.random.randn(128, 768).astype(np.float32)
q_int8, scale = quantize_symmetric(weights, num_bits=8)
recon = q_int8.astype(np.float64) * scale
print("INT8 error:", quantization_error(weights, recon))

```

Per-channel INT8 for improved matrix quality:

```python
from quantization import quantize_per_channel

q_pc, scales = quantize_per_channel(weights, num_bits=8, axis=0)
recon_pc = q_pc.astype(np.float64) * scales[:, None]
print("Per-channel INT8 error:", quantization_error(weights, recon_pc))

```

Asymmetric quantization with zero-point for non-negative activations:

```python
from quantization import quantize_asymmetric

activations = np.abs(np.random.randn(256, 1024)).astype(np.float32)
q_asym, scale_a, zp = quantize_asymmetric(activations, num_bits=8)
recon_a = (q_asym - zp) * scale_a
print("Asymmetric INT8 error:", quantization_error(activations, recon_a))

```

Comparing all methods side-by-side:

```python
from quantization import compare_quantization_methods

compare_quantization_methods(weights, num_bits=8)

```

This outputs a comparison table:

```

  Method               MSE   SNR (dB)   Cosine Sim   Max Error
  ------------------------------------------------------------
  Per-tensor sym     0.00023      38.1       0.99998      0.0041
  Per-channel sym    0.00012      44.3       0.99999      0.0027
  Asymmetric         0.00017      40.9       0.99999      0.0033

```

## Summary

- **INT8** provides the best performance-to-simplicity ratio for GPU servers, offering 2× throughput with ~0.5% quality loss via per-tensor symmetric quantization.
- **GPTQ** enables 4× compression through Hessian-based selective bit allocation, making 70B model inference feasible on consumer GPUs with minimal perplexity degradation.
- **AWQ** accelerates GPTQ by 1.5–2× through activation-aware weight protection, trading some calibration complexity for faster inference.
- **GGUF** packages mixed-precision models into portable files for CPU-only edge deployment, prioritizing accessibility over raw throughput.
- The reference implementations in [`main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/main.py) and [`main.rs`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/main.rs) demonstrate that these algorithms can be built from scratch using only NumPy or Rust's stdlib, demystifying the black-box nature of production quantization libraries.

## Frequently Asked Questions

### What is the difference between GPTQ and AWQ quantization?

**GPTQ** uses a Hessian matrix computed from calibration data to identify and protect sensitive weights during INT4 quantization, optimizing for mathematical precision. **AWQ** achieves similar compression by identifying the 1% of weights that interact with large activations and scaling them before quantization, resulting in 1.5–2× faster inference than GPTQ with comparable quality. Both are post-training quantization methods requiring calibration data, but AWQ is generally preferred for latency-sensitive applications.

### When should I use INT8 versus INT4 quantization?

Use **INT8** when deploying to GPU servers (NVIDIA A100, H100) where 8-bit integer arithmetic is natively accelerated and you need the highest throughput with minimal accuracy loss (~0.5%). Use **INT4** (via GPTQ or AWQ) when memory constraints are severe—such as running 70B models on 24GB consumer GPUs—where the 4× compression justifies the 1–2% perplexity degradation. INT8 requires less calibration and is more robust across different model architectures.

### What is GGUF format used for?

**GGUF** (GPT-Generated Unified Format) is a portable container format used by [`llama.cpp`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/llama.cpp) to store mixed-precision quantized models (e.g., Q4_K_M) along with metadata. It enables large language models to run on edge devices, laptops, and CPUs (including Apple Silicon and ARM) without requiring CUDA. While inference is slower than native GPU quantization, GGUF makes 70B-parameter models accessible to users without datacenter hardware.

### How does the ai-engineering-from-scratch implementation differ from production libraries?

The repository implements quantization algorithms in **pure Python/NumPy** (and a Rust stdlib-only version) to teach the underlying mathematics without dependencies on PyTorch or TensorRT. Unlike production libraries such as `vLLM` or `auto-gptq`, which optimize for speed through CUDA kernels and C++ extensions, the [`main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/main.py) implementation focuses on clarity and educational value, providing bit-level converters and explicit error metrics (MSE, SNR, cosine similarity) that are often hidden in production code.