# How to Reduce Inference Cost with Quantization: A Complete Technical Guide

> Reduce inference cost with quantization. Learn how INT8 and 4-bit models cut memory by 75% and boost throughput 2-4x. Optimize models without code changes.

- Repository: [Chip Huyen/aie-book](https://github.com/chiphuyen/aie-book)
- Tags: how-to-guide
- Published: 2026-04-24

---

**Quantization reduces inference cost by compressing model weights from FP32/FP16 to lower-bit integers like INT8 or 4-bit, cutting memory usage by up to 75% and increasing throughput 2-4× via optimized integer kernels without requiring changes to model architecture or serving code.**

The *AI Engineering* book by Chip Huyen (repository `chiphuyen/aie-book`) identifies quantization as one of the highest-impact techniques to reduce inference cost at scale. According to the source materials located at [`chapter-summaries.md`](https://github.com/chiphuyen/aie-book/blob/main/chapter-summaries.md) (lines 206-213) and listed as a dedicated subsection in [`ToC.md`](https://github.com/chiphuyen/aie-book/blob/main/ToC.md) (line 126), quantization delivers immediate hardware efficiency gains while maintaining model accuracy. This guide explains how to implement quantization workflows to slash serving expenses.

## What Is Model Quantization?

Quantization is a **model-level optimization** that compresses a neural network's weights (and optionally activations) from high-precision floating-point formats—typically FP32 or FP16—to lower-bit integer representations such as **INT8** or **4-bit**. As documented in the repository, this technique does not alter the model architecture, meaning your existing inference service code can remain unchanged; only the model artifact requires replacement.

## Why Quantization Cuts Inference Costs

Moving from FP32 to INT8 delivers three measurable gains that directly reduce inference cost:

- **Memory footprint shrinks** — Models store and load with approximately ¼ the original size when converting from FP32 to INT8.
- **Bandwidth and cache efficiency improve** — Smaller tensors fit better in GPU and CPU caches, reducing memory-access latency.
- **Compute accelerates** — Modern AI accelerators—including NVIDIA Tensor Cores, AMD MI series, and Intel Gaudi—expose integer-matrix-multiply instructions that execute 2-4× more operations per clock cycle than FP32.

## Choosing a Quantization Workflow

The `chiphuyen/aie-book` analysis outlines two primary approaches to quantization:

**Post-Training Quantization (PTQ)**

PTQ requires no retraining or additional data. It works by calibrating activation statistics on a small representative dataset, then converting weights to INT8. This is the fastest path to reduce inference cost and typically sacrifices less than 1% perplexity on language models.

**Quantization-Aware Training (QAT)**

QAT integrates quantization constraints during the fine-tuning phase, resulting in higher accuracy than PTQ but requiring computational resources for additional training epochs.

### The Five-Step Quantization Pipeline

According to the repository's technical summaries, a complete workflow involves:

1. **Export the model** — Convert checkpoints to framework-agnostic formats like ONNX or Hugging Face `PreTrainedModel`.
2. **Choose a scheme** — Select PTQ for speed or QAT for accuracy preservation.
3. **Calibrate (PTQ only)** — Run a representative dataset through `torch.quantization.Calibrator` to collect activation statistics for scaling.
4. **Convert** — Apply integer kernels using tools like `optimum` for ONNX Runtime or `accelerate` for Hugging Face, rewriting the graph to use INT8 or 4-bit operations.
5. **Deploy** — Swap the FP32 model file with the quantized artifact; serving stacks like vLLM, Triton, or TensorRT handle the rest without code changes.

## Practical Implementation: 8-Bit Quantization with BitsAndBytes

Below is a complete, runnable example that quantizes a Hugging Face Llama-2 model to 8-bit using the `bitsandbytes` library—a popular PTQ approach referenced in the book's [`resources.md`](https://github.com/chiphuyen/aie-book/blob/main/resources.md) and demonstrated in [`case-studies.md`](https://github.com/chiphuyen/aie-book/blob/main/case-studies.md).

```python

# quantize_llama.py

from transformers import AutoModelForCausalLM, AutoTokenizer
import bitsandbytes as bnb

# 1️⃣ Load the FP16 model (the original checkpoint)

model_name = "meta-llama/Llama-2-7b-hf"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",          # load across GPUs if available

    torch_dtype="auto",         # keep FP16 for the original

)

# 2️⃣ Apply 8‑bit quantization (post‑training)

quantized_model = bnb.nn.Int8Params.from_pretrained(
    model,
    # The following flag tells bitsandbytes to replace linear layers with INT8 kernels.

    # It works without any calibration data.

    replace_linear=True,
)

# 3️⃣ Save the quantized checkpoint (can be used by any serving stack)

quantized_model.save_pretrained("./llama2-7b-8bit")
tokenizer.save_pretrained("./llama2-7b-8bit")

# 4️⃣ Quick inference check

from transformers import pipeline

generator = pipeline(
    "text-generation",
    model="./llama2-7b-8bit",
    tokenizer=tokenizer,
    device_map="auto",
    torch_dtype="auto",
)

print(generator("Explain quantization in a sentence:", max_new_tokens=30)[0]["generated_text"])

```

**Key Implementation Details**

- `bitsandbytes.nn.Int8Params.from_pretrained` automatically swaps every linear layer for an INT8 implementation and handles scaling factors internally.
- No calibration dataset is required, making this ideal for rapid cost-reduction experiments.
- The saved `./llama2-7b-8bit` directory can be mounted into a Triton model repository or served with vLLM without any code modifications.

## Deploying Quantized Models with vLLM

To realize the cost savings in production, serve the quantized artifact using **vLLM**, which automatically detects INT8 kernels and optimizes GPU memory allocation.

```bash

# launch vLLM with the quantized model

vllm serve ./llama2-7b-8bit \
    --model-name llama2-7b-8bit \
    --dtype float16   # model weights are already INT8, dtype flag is a no‑op

```

vLLM will run inference up to **2-3× faster** compared with the FP16 baseline while cutting GPU memory usage by roughly **75%**, directly translating to lower cloud instance costs.

## Summary

- **Quantization compresses weights** from FP32/FP16 to INT8 or 4-bit, reducing model size by approximately 75%.
- **Post-training quantization (PTQ)** offers the fastest path to reduce inference cost without retraining or calibration data, typically sacrificing less than 1% accuracy.
- **Hardware acceleration** on modern GPUs (A100, H100, T4) and CPUs provides 2-4× throughput improvements via optimized integer kernels.
- **Zero code changes** are required for deployment; the quantized model artifact drops into existing serving stacks like vLLM, Triton Inference Server, and TorchServe.

## Frequently Asked Questions

### Does quantization work for all model architectures?

Yes. Quantization is model-agnostic and works for transformers, diffusion models, and vision encoders alike. According to the `chiphuyen/aie-book` analysis, the same quantization principles apply across architectures because they operate on the numerical representation of weights rather than model structure.

### How much accuracy do I lose with INT8 post-training quantization?

INT8 PTQ typically loses less than 1% perplexity on language models, making it suitable for production use. If you require higher fidelity, quantization-aware training (QAT) can recover most of the loss by fine-tuning with simulated quantization during training.

### Can I use quantized models with any inference server?

Most modern serving stacks support quantized models out-of-the-box. The `chiphuyen/aie-book` repository notes that vLLM, Triton Inference Server, and TorchServe all include optimized INT8 kernels. You simply replace the FP32 model file with the quantized checkpoint; no changes to inference code are required.

### What is the difference between 8-bit and 4-bit quantization?

8-bit (INT8) quantization reduces model size by 75% and is widely supported by hardware accelerators. 4-bit quantization achieves even higher compression (87.5% reduction) but may require specific kernel support and can incur slightly higher accuracy trade-offs. Both methods are covered in the book's resources for reducing inference cost.