# How the AI Engineering from Scratch Curriculum Addresses Production Concerns for AI Models: Quantization and Inference Optimization

> Learn how the AI Engineering from Scratch curriculum tackles AI model production issues like quantization and inference optimization. Explore efficient deployment strategies.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: deep-dive
- Published: 2026-06-23

---

*Note: This article references specific file paths from the rohitg00/ai-engineering-from-scratch repository.*

# How the AI Engineering from Scratch Curriculum Addresses Production Concerns for AI Models: Quantization and Inference Optimization

**The curriculum treats production-grade AI modeling as a first-class learning objective by guiding learners through hardware-aware engine selection, six production-ready quantization formats, and KV-cache optimization to transform research models into deployable services.**

The **AI Engineering from Scratch** curriculum addresses production concerns for AI models through a dedicated infrastructure track located in `phases/17-infrastructure-and-production/`. By treating quantization and inference optimization as core competencies rather than afterthoughts, the course teaches learners to navigate hardware constraints, calibration datasets, and latency metrics using actual source files and benchmarking tools.

## Selecting the Right Inference Engine for Your Hardware

The curriculum begins production planning with hardware-aware engine selection. In [`phases/17-infrastructure-and-production/28-self-hosted-serving-selection/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/17-infrastructure-and-production/28-self-hosted-serving-selection/docs/en.md), learners compare four dominant inference engines—**llama.cpp**, **Ollama**, **vLLM**, and **SGLang**—and map them to specific hardware tiers:

- **CPU/edge**: Optimized for lightweight deployments
- **AMD GPUs**: Compatibility-focused configurations
- **NVIDIA GPUs**: Standard data center acceleration
- **Blackwell/Hopper GPUs**: Next-generation NVIDIA architectures

This mapping guarantees that the selected engine actually runs on the target hardware, avoiding costly "engine-incompatible" failures before quantization even begins.

## Production-Grade Quantization Strategies

The [`phases/17-infrastructure-and-production/09-production-quantization/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/17-infrastructure-and-production/09-production-quantization/docs/en.md) lesson treats quantization as a systematic decision rather than a binary compression choice. It details bit-width trade-offs, quality tolerance thresholds, and engine compatibility matrices.

### The Six Production-Ready Formats

The curriculum defines six production-grade quantization schemes with distinct sweet spots:

1. **GGUF Q4_K_M / Q5_K_M**: Optimized for CPU and edge devices using llama.cpp; Q4_K_M for aggressive compression, Q5_K_M for balanced quality.
2. **GPTQ**: Ideal for vLLM deployments requiring **multi-LoRA** (Low-Rank Adaptation) serving.
3. **AWQ**: Recommended for vLLM when model size is ≤7B and workload involves routine chat scenarios.
4. **FP8**: Used with vLLM or SGLang on H100/Blackwell GPUs when quality tolerance is high.
5. **MXFP4**: For bleeding-edge throughput on supported NVIDIA hardware.
6. **NVFP4**: NVIDIA-specific 4-bit format for maximum memory reduction on compatible Blackwell/Hopper GPUs.

### Avoiding the Calibration-Dataset Trap

The curriculum explicitly warns against the **calibration-dataset trap**—using out-of-domain data for quantization calibration, which silently destroys model capabilities. The lesson mandates domain-specific validation and ties quantization selection to downstream evaluation suites including HumanEval, MATH, and MMLU.

## Edge Inference and Memory Constraints

Deploying to constrained environments requires additional safeguards. The [`phases/17-infrastructure-and-production/12-edge-inference/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/17-infrastructure-and-production/12-edge-inference/docs/en.md) lesson pairs hardware platforms with appropriate quantization:

- **Apple ANE**: Core ML INT4+FP16
- **Qualcomm Hexagon**: QNN INT8/INT4
- **WebGPU**: Q4 formats
- **Jetson**: NVFP4

The lesson formalizes a **budget check** comparing model size against device RAM plus KV-cache footprint, preventing OOM crashes before deployment.

## Optimizing Inference Throughput and Latency

Production optimization extends beyond model weights to serving infrastructure.

### Measuring Goodput with TPOT and ITL

According to [`phases/17-infrastructure-and-production/08-inference-metrics-goodput/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/17-infrastructure-and-production/08-inference-metrics-goodput/docs/en.md), the curriculum defines precise latency metrics:

- **TPOT** (Time Per Output Token): Measures generation speed after the first token
- **ITL** (Inter-Token Latency): Tracks consistency between consecutive tokens

These metrics provide quantitative footing for comparing quantized versus full-precision serving in production streams.

### Multi-Region KV Cache Locality

The [`phases/17-infrastructure-and-production/11-multi-region-kv-locality/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/17-infrastructure-and-production/11-multi-region-kv-locality/docs/en.md) lesson explains that naive round-robin load balancing destroys cached prefixes, inflating TTFT (Time To First Token). The curriculum prescribes **KV-aware routers** such as vLLM Router and **llm-d** to maintain cache locality across regions, saving tens to hundreds of milliseconds per request.

## Practical Implementation: The Quantization Picker

The curriculum includes a practical skill output in [`phases/17-infrastructure-and-production/09-production-quantization/outputs/skill-quantization-picker.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/17-infrastructure-and-production/09-production-quantization/outputs/skill-quantization-picker.md) that converts theoretical knowledge into actionable deployment artifacts. The picker accepts hardware type, engine, model size, workload, and quality tolerance as inputs, then returns a concrete quantization format plus a calibration-validation plan.

```python
import json
from pathlib import Path

# Load the skill output (already generated by the lesson)

skill_path = Path(
    "phases/17-infrastructure-and-production/09-production-quantization/"
    "outputs/skill-quantization-picker.md"
)
skill = json.loads(skill_path.read_text())

def pick_quantization(hw, engine, model_size, workload, quality_tol):
    # Simplified decision matrix mirrors the lesson's guidance

    if hw == "CPU":
        return "GGUF Q4_K_M"
    if engine == "vLLM" and workload == "multi-loRA":
        return "GPTQ"
    if engine == "vLLM" and model_size <= 7 and workload == "routine chat":
        return "AWQ"
    if engine in ("vLLM", "SGLang") and quality_tol == "high":
        return "FP8"
    return "NVFP4"

# Example call for H100 deployment

chosen = pick_quantization(
    hw="GPU-H100", engine="vLLM", model_size=70,
    workload="reasoning", quality_tol="high"
)
print("Chosen quantization format:", chosen)

# Mock validation: run a tiny benchmark (placeholder)

def mock_throughput(format_name):
    base = {"GGUF Q4_K_M": 550, "AWQ": 740, "GPTQ": 710,
            "FP8": 820, "NVFP4": 860, "MXFP4": 850}
    return base.get(format_name, 500)

print("Estimated tok/s:", mock_throughput(chosen))

```

The output includes a **calibration-dataset checklist** and a **rollback plan** specifying when to revert to BF16 if quality degradation exceeds tolerance.

## Validating Production Readiness

Capstone projects enforce end-to-end validation. In [`phases/19-capstone-projects/14-speculative-decoding-server/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/14-speculative-decoding-server/docs/en.md), learners must benchmark throughput, tail-latency, and cost ($/1M tokens) for each quantization choice. Additionally, the [`phases/17-infrastructure-and-production/09-production-quantization/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/17-infrastructure-and-production/09-production-quantization/code/main.py) utility computes weight, KV, and activation footprints for each format, enabling pre-deployment memory verification.

## Summary

- **Hardware-aware engine selection**: The curriculum maps four inference engines to specific hardware tiers in [`phases/17-infrastructure-and-production/28-self-hosted-serving-selection/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/17-infrastructure-and-production/28-self-hosted-serving-selection/docs/en.md) to prevent incompatible deployments.
- **Six quantization formats**: Production-ready options include GGUF Q4_K_M, GPTQ, AWQ, FP8, MXFP4, and NVFP4, each with defined bit-widths and engine compatibility matrices.
- **Calibration validation**: The [`skill-quantization-picker.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/skill-quantization-picker.md) output requires a domain-specific calibration dataset and includes a rollback plan to prevent quality degradation on benchmarks like HumanEval and MMLU.
- **Edge constraints**: Memory budgeting (model size vs. device RAM) is enforced in the edge inference lesson to prevent OOM crashes on Apple ANE, Qualcomm Hexagon, and Jetson devices.
- **Performance metrics**: Goodput measurement uses TPOT (Time Per Output Token) and ITL (Inter-Token Latency) to compare quantized versus full-precision serving.
- **Cache-aware routing**: Multi-region deployments leverage KV-aware routers like vLLM Router and llm-d to maintain low TTFT (Time To First Token).

## Frequently Asked Questions

### What are the six production-grade quantization formats taught in the curriculum?

The curriculum covers **GGUF Q4_K_M** and **Q5_K_M** for CPU/edge deployment with llama.cpp, **GPTQ** and **AWQ** for GPU serving with vLLM, **FP8** for high-quality tolerance on H100/Blackwell GPUs, and **NVFP4**/**MXFP4** for bleeding-edge NVIDIA hardware. Each format is documented in [`phases/17-infrastructure-and-production/09-production-quantization/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/17-infrastructure-and-production/09-production-quantization/docs/en.md) with specific bit-widths, compatible engines, and quality trade-offs.

### How does the curriculum prevent quality degradation when quantizing models?

The curriculum emphasizes the **calibration-dataset trap**, warning against using out-of-domain data for calibration. The [`skill-quantization-picker.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/skill-quantization-picker.md) artifact requires learners to specify a domain-specific validation plan and includes a rollback mechanism to revert to BF16 if quality drops below tolerance thresholds during evaluation on benchmarks like HumanEval or MMLU.

### What metrics does the curriculum use to evaluate inference optimization?

According to [`phases/17-infrastructure-and-production/08-inference-metrics-goodput/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/17-infrastructure-and-production/08-inference-metrics-goodput/docs/en.md), the curriculum teaches **goodput** measurement using **TPOT** (Time Per Output Token) and **ITL** (Inter-Token Latency). These metrics provide a quantitative footing for comparing quantized against full-precision models in production streams.

### How does the curriculum address multi-region deployment latency?

The [`phases/17-infrastructure-and-production/11-multi-region-kv-locality/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/17-infrastructure-and-production/11-multi-region-kv-locality/docs/en.md) lesson explains that naive round-robin load balancing destroys cached prefixes, inflating TTFT. Instead, the curriculum prescribes **KV-aware routers** such as vLLM Router and llm-d to maintain cache locality, reducing TTFT penalties by tens to hundreds of milliseconds per request across distributed regions.