Techniques for Quantizing Deep Learning Models: A Complete Guide to INT8 Implementation in tinytorch

The harvard-edge/cs249r_book repository implements symmetric per-tensor INT8 quantization, full model post-training quantization, and de-quantization utilities in the tinytorch framework, enabling 4× model compression with minimal accuracy loss.

Deep learning model quantization converts 32-bit floating-point parameters to lower-precision integers, drastically reducing memory bandwidth and computational requirements for edge deployment. The harvard-edge/cs249r_book repository demonstrates production-ready quantization techniques through its tinytorch educational framework. This guide examines the specific implementations, source code locations, and practical applications of these compression methods.

Symmetric Per-Tensor INT8 Quantization

The foundation of the quantization pipeline is symmetric per-tensor INT8 quantization, which scales entire weight tensors to the signed 8-bit integer range [-128, 127]. This technique uses a single scale factor and zero-point applied uniformly across the tensor, making it the simplest and most hardware-compatible quantization scheme.

In tinytorch/src/15_quantization/15_quantization.py, the quantize_int8 function (lines 48-62) implements this approach by calculating the optimal scale factor to map the tensor's dynamic range to the INT8 bounds. The function returns the quantized tensor alongside its metadata (scale and zero-point), which are required for faithful reconstruction during inference.

Post-Training Model Quantization Pipeline

For production deployment, the repository provides post-training quantization through the Quantizer.quantize_model class method (lines 29-43 in tinytorch/src/15_quantization/15_quantization.py). This utility automatically traverses every layer of a model, applies INT8 quantization to parameters, and generates a comprehensive statistics dictionary.

The method reports:

• Original size (MiB) versus quantized size (MiB)
• Compression ratio achieved across the model
• Per-parameter statistics including individual scale factors and zero-points for each layer

This automated pipeline requires no retraining or architecture modifications, making it ideal for rapid deployment of pre-trained models on resource-constrained devices.

De-quantization and Inference Utilities

To support inference with quantized models, the repository implements de-quantization via the dequantize_int8 function (lines 5-15 in tinytorch/src/15_quantization/15_quantization.py). This utility reverses the quantization operation by applying the inverse scaling formula, recovering a float tensor from its INT8 representation.

For efficient inference, the codebase also provides the QuantizedLinear layer (exported at line 51). This wrapper stores quantized weights and biases, performing on-the-fly de-quantization during forward passes. The design minimizes memory usage while maintaining compatibility with standard tinytorch model architectures.

Quantization-Aware Training Support

While the current implementation focuses on post-training methods, the repository includes scaffolding for Quantization-Aware Training (QAT). The Quantizer class structure (referenced at line 37 in the module header) is designed to support future QAT extensions, where fake quantization nodes would simulate INT8 arithmetic during training to improve accuracy.

This placeholder infrastructure allows developers to extend the existing quantization pipeline with per-channel scaling or mixed-precision schemes without refactoring the core API.

Source Code Structure and Implementation Details

The quantization system is organized across several key files in the repository:

• tinytorch/src/15_quantization/15_quantization.py – Core implementation containing quantize_int8, dequantize_int8, the Quantizer class, and QuantizedLinear definitions
• tinytorch/perf/__init__.py – Package-level exports enabling imports via tinytorch.perf.quantization
• tinytorch/tests/15_quantization/test_quantizer_core.py – Unit tests verifying quantization correctness, range validation, and error tolerance
• tinytorch/tests/15_quantization/test_quantization_integration.py – Integration tests validating end-to-end model quantization round-trips

Practical Code Examples

The following examples demonstrate the complete quantization workflow using the tinytorch API:

# Basic per-tensor INT8 quantization

from tinytorch.perf.quantization import quantize_int8, dequantize_int8
from tinytorch.tensor import Tensor

fp32 = Tensor([[0.5, -1.2, 2.3], [1.0, 0.0, -0.7]])
q_tensor, scale, zp = quantize_int8(fp32)  # Lines 48-62

print("INT8 tensor:", q_tensor.data)
print("scale / zero-point:", scale, zp)

# De-quantization back to FP32

restored = dequantize_int8(q_tensor, scale, zp)  # Lines 5-15

print("Recovered FP32:", restored.data)

# Full model quantization (post-training)

from tinytorch.perf.quantization import Quantizer
from my_model import MyModel

model = MyModel()
quant_info = Quantizer.quantize_model(model)  # Lines 29-43

print("Original size (MiB):", quant_info["original_size_mb"])
print("Quantized size (MiB):", quant_info["quantized_size_mb"])
print("Compression ratio:", quant_info["compression_ratio"])

# Inspect per-layer statistics

for name, stats in quant_info["quantized_layers"].items():
    print(f"{name}: scale={stats['scale']:.4f}, zp={stats['zero_point']}")

Summary

• Symmetric per-tensor INT8 quantization in quantize_int8 (lines 48-62) provides the core 8-bit compression mechanism using single-scale quantization.
• Post-training quantization via Quantizer.quantize_model (lines 29-43) automates the conversion of entire models without requiring retraining.
• De-quantization utilities and QuantizedLinear layers enable efficient inference with compressed representations.
• The repository structure supports future quantization-aware training extensions through the existing Quantizer class scaffolding.
• All implementations mirror PyTorch's torch.quantization API patterns, facilitating migration to production frameworks.

Frequently Asked Questions

What is the difference between post-training quantization and quantization-aware training?

Post-training quantization converts a trained FP32 model to INT8 weights after training completes, using calibration data to determine optimal scale factors. Quantization-aware training simulates low-precision arithmetic during the training process itself, allowing the model to adapt to quantization noise and typically achieving higher accuracy. The cs249r_book repository currently implements post-training methods while providing infrastructure for future QAT support.

How much memory reduction does INT8 quantization provide?

INT8 quantization reduces model size by approximately 4× compared to 32-bit floating-point representations, since 8 bits replace 32 bits per parameter. The Quantizer.quantize_model method reports exact compression ratios in the compression_ratio field of its returned dictionary.

Does tinytorch support per-channel quantization?

The current implementation focuses on per-tensor quantization, applying a single scale factor across entire weight tensors. However, the Quantizer class architecture is designed to accommodate per-channel extensions, which would store separate scale factors for each output channel in convolutional or linear layers.

Where is the quantization implementation located in the repository?

All core quantization logic resides in tinytorch/src/15_quantization/15_quantization.py within the harvard-edge/cs249r_book repository. This file contains the quantize_int8 and dequantize_int8 functions, the Quantizer class for model-wide quantization, and the QuantizedLinear layer definition for inference.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}