Quantization Methods for Reducing LLM Model Size: 5 Techniques Explained

Quantization compresses large language models by converting 32-bit floating point weights to lower-precision formats such as INT8 or FP16, reducing memory footprint by up to 75% while preserving model capability through post-training calibration or training-aware optimization.

According to the AI Engineering book repository by Chip Huyen (chiphuyen/aie-book), quantization stands as one of the most impactful model-agnostic techniques for inference-level optimization. This approach reduces the numeric precision of weights and activations to shrink model size and accelerate computation without altering the underlying transformer architecture.

What Is Quantization in LLMs?

Quantization is a compression technique applied after pre-training that lowers the bit-width of a model's parameters. As documented in chapter-summaries.md (lines 206–213), quantization works "across models" to reduce latency and enable serving larger models on limited hardware.

The process targets specific architectural components:

• Embedding layers – Weights are statically quantized to reduce the initial memory load
• Transformer blocks – Both weights and activations are quantized to accelerate matrix multiplications in self-attention and feed-forward networks
• LayerNorm and Softmax – Typically retained in FP32 for numerical stability during normalization and probability calculation

Types of Quantization Methods

The repository outlines five primary families of quantization, each suited to different deployment constraints and accuracy requirements.

Post-Training Static Quantization (PTQ)

PTQ calibrates the model once using a representative dataset to compute optimal scaling factors. Weights are then quantized permanently to INT8 or lower.

• Best for: Rapid deployment when fine-tuning resources are unavailable
• Trade-off: Fastest implementation but potential minor accuracy degradation

Post-Training Dynamic Quantization (PTDQ)

PTDQ quantizes only the weight tensors ahead of time while keeping activations in higher precision (FP16 or FP32). Activations are quantized on-the-fly during inference.

• Best for: Low-latency CPU inference where model size matters most
• Advantage: Avoids calibration data requirements

Quantization-Aware Training (QAT)

QAT introduces simulated quantization operations during the training or fine-tuning phase. This allows the model to learn weight distributions that compensate for precision loss.

• Best for: Maximum accuracy with aggressive bit-width reduction (e.g., 4-bit)
• Requirement: Full training pipeline access

Mixed-Precision Schemes

Mixed-precision assigns different bit-widths to different layers based on sensitivity analysis. For example, attention layers might use FP16 while feed-forward blocks use INT8.

• Best for: Large models on limited GPU memory (e.g., running LLaMA-13B on a single consumer GPU)
• Implementation: Often combined with 4-bit or 8-bit kernels

Weight-Only Quantization

Weight-only quantization compresses only the static weight tensors while leaving activations in FP16. Modern GPUs with tensor cores can process low-precision weights efficiently while maintaining FP16 activation precision.

• Best for: Modern NVIDIA GPUs with bfloat16/fp16 tensor core support
• Benefit: Optimal balance between memory savings and computational stability

Practical Implementation with Python

The following example demonstrates 8-bit weight-only quantization using the bitsandbytes library, compatible with any Hugging Face transformer model. This implementation reflects the practical approach emphasized in the repository's model-level optimization discussions.

# Install dependencies:

# pip install transformers bitsandbytes accelerate

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# Select a model for demonstration

model_name = "EleutherAI/gpt-neo-125M"

# Load tokenizer (unaffected by quantization)

tokenizer = AutoTokenizer.from_pretrained(model_name)

# Load model with 8-bit weight quantization

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",          # Automatically distributes layers across devices

    load_in_8bit=True,         # Enable 8-bit weight-only quantization

    torch_dtype=torch.float16  # Keep activations in FP16 for speed

)

# Prepare input prompt

prompt = "Explain why quantization is useful for large language models."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

# Generate response

with torch.no_grad():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=100,
        temperature=0.7,
        do_sample=True
    )

print(tokenizer.decode(output_ids[0], skip_special_tokens=True))

Key parameters explained:

• load_in_8bit=True triggers the quantization engine to compress weights from FP32 to INT8
• device_map="auto" enables efficient layer placement for limited GPU memory, directly supporting the book's goal of "reducing latency and enabling serving larger models"
• torch_dtype=torch.float16 maintains mixed-precision for activations, preventing numerical instability in the softmax and layer normalization layers

Repository Structure and Key Files

The chiphuyen/aie-book repository organizes its quantization coverage across these critical files:

• chapter-summaries.md (lines 206–213) – Explicitly identifies quantization as a primary model-agnostic optimization technique alongside distillation
• ToC.md (line 126) – Lists a dedicated Quantization section in the table of contents, indicating comprehensive coverage of the topic
• README.md – Provides navigation to chapters discussing quantization trade-offs and hardware-specific implementations

Summary

• Quantization reduces LLM model size by lowering weight precision from FP32 to INT8, FP16, or 4-bit formats without architectural changes
• PTQ offers the fastest deployment path, while QAT maximizes accuracy for aggressive compression
• Weight-only quantization provides the best trade-off for modern GPU inference, keeping activations in FP16
• The bitsandbytes library enables 8-bit quantization via load_in_8bit=True and device_map="auto" for automatic memory management
• According to the source code in chapter-summaries.md, quantization "generally works well across models," making it a universal first-step optimization

Frequently Asked Questions

How much memory can quantization actually save?

Converting from FP32 to INT8 reduces model size by approximately 75%, while 4-bit quantization achieves up to 87.5% compression. For a 13B parameter model, this means reducing VRAM requirements from roughly 52 GB to 13 GB (INT8) or 6.5 GB (4-bit), enabling deployment on consumer hardware.

What is the difference between dynamic and static post-training quantization?

Static quantization (PTQ) computes fixed scaling factors using calibration data before deployment, permanently converting weights to lower precision. Dynamic quantization (PTDQ) quantizes weights ahead of time but converts activations on-the-fly during inference, eliminating the need for calibration data but potentially introducing slight latency overhead.

Does quantization require retraining the model?

Post-training methods (PTQ and PTDQ) do not require retraining or fine-tuning, making them accessible for production models. However, Quantization-Aware Training requires integrating simulated quantization into the training loop to maintain accuracy with aggressive bit reduction.

Which quantization method is best for production LLMs?

For GPU production environments, weight-only quantization (8-bit or 4-bit) using libraries like bitsandbytes or AutoGPTQ is currently optimal. This approach maximizes memory reduction while preserving FP16 activation precision, maintaining model quality with minimal latency impact.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}