# How to Optimize Model Size with LiteRT: Quantization and External Weights Explained

> Optimize model size with LiteRT using quantization and external weights. Reduce footprints up to 4x or separate large weights from metadata.

- Repository: [google-ai-edge/LiteRT](https://github.com/google-ai-edge/LiteRT)
- Tags: how-to-guide
- Published: 2026-03-13

---

**Post-training weight quantization and external weight files are the two primary methods to optimize model size with LiteRT, reducing model footprints by up to 4× or separating gigabyte-scale weights from the flatbuffer metadata.**

The google-ai-edge/LiteRT repository provides a lightweight runtime for on-device inference, but deploying large neural networks on edge devices requires aggressive footprint reduction. This guide demonstrates how to optimize model size with LiteRT using post-training quantization and external weight loading, referencing the actual source implementation in `tflite/tools/optimize/quantize_model.cc` and [`weight_loader/external_weight_loader_litert.h`](https://github.com/google-ai-edge/LiteRT/blob/main/weight_loader/external_weight_loader_litert.h).

## Post-Training Weight Quantization

**Post-training weight quantization** converts 32-bit floating-point weights to 8-bit integers (INT8), reducing the model size to approximately one-quarter of the original. This technique preserves the original graph structure while enabling **hybrid kernels** that keep activations in float32 for inference stability.

The quantization transformation is implemented in the `QuantizeWeights` function within `tflite/tools/optimize/quantize_model.cc`. It walks the model graph and inserts `Quantize` and `Dequantize` operations only where necessary, ensuring compatibility with LiteRT's hybrid execution mode.

### Command-Line Conversion

The simplest way to apply weight quantization is via the `tflite_convert` CLI tool. Use the `--post_training_quantize` flag to invoke the optimizer pipeline:

```bash
tflite_convert \
  --output_file=/tmp/model_quant.tflite \
  --saved_model_dir=/tmp/saved_model \
  --post_training_quantize

```

This command processes the SavedModel, applies the INT8 transformation to all eligible weight tensors, and outputs a flatbuffer containing quantized values. The resulting model runs on LiteRT's hybrid kernels without requiring additional application code changes.

### Programmatic C++ API

For custom quantization parameters or integration into existing build pipelines, call the optimizer directly from C++. The following example loads a model flatbuffer, quantizes the weights, and retrieves the new buffer:

```cpp
#include "tflite/tools/optimize/quantize_model.h"
#include "flatbuffers/flatbuffer_builder.h"
#include "tflite/schema/schema_generated.h"

int main() {
  // Load the original model flatbuffer.
  const tflite::Model* model = ::tflite::GetModel(original_buffer);
  flatbuffers::FlatBufferBuilder builder;

  // Quantize the whole model (weights only). Activation type stays FLOAT.
  TfLiteStatus status = ::tflite::optimize::QuantizeWeights(&builder, model);
  if (status != kTfLiteOk) {
    fprintf(stderr, "Quantization failed!\n");
    return -1;
  }

  // Obtain the new model buffer.
  const uint8_t* quantized_buf = builder.GetBufferPointer();
  size_t quantized_size = builder.GetSize();
  // ... write `quantized_buf` to disk or hand it to LiteRT runtime.
}

```

The `QuantizeWeights` function handles the complexity of determining per-tensor quantization parameters and updating the schema buffers accordingly.

## External Weight Files for Large Models

When model weights exceed the 2 GB flatbuffer limit or you need to minimize download sizes for over-the-air updates, **external weight files** store large weight blobs outside the `.tflite` file. Only metadata remains inside the flatbuffer, often reducing the model file to under 1 MB while the actual weights reside in separate binary files.

This functionality is exposed through the [`weight_loader/external_weight_loader_litert.h`](https://github.com/google-ai-edge/LiteRT/blob/main/weight_loader/external_weight_loader_litert.h) header, with the concrete implementation in `weight_loader/external_weight_loader_litert.cc`.

### Runtime Loading Implementation

To load external weights at runtime, create a `WeightLoader` instance and map the external files into tensor buffers:

```cpp
#include "weight_loader/external_weight_loader_litert.h"

int main() {
  // 1️⃣ Load the flatbuffer (contains only metadata)
  const tflite::Model* model = ::tflite::GetModel(model_buffer);

  // 2️⃣ Create a WeightLoader that knows where the files live.
  //    `model_dir` points to the folder containing the *.weight* files.
  auto loader = weight_loader::CreateLiteRtWeightLoader(
      model, std::optional<std::string>("/path/to/weights/"));

  // 3️⃣ Prepare CPU access (or OpenCL if you have a GPU).
  weight_loader::WeightAccessRequest req;
  req.cpu = true;            // load into host RAM
  req.opencl = false;        // no GPU copy yet
  loader->PrepareAccess(req, nullptr /*environment*/);

  // 4️⃣ For each external buffer, map the file and give the runtime a tensor buffer.
  for (const auto& info : loader->GetWeightInfo()) {
    const weight_loader::WeightInfo* w = loader->FindWeightInfoByBuffer(info.external_buffer_id);
    // Load the raw bytes from disk (simplified):
    std::vector<uint8_t> data(w->length);
    std::ifstream file(std::to_string(info.external_buffer_id) + ".weight",
                       std::ios::binary);
    file.read(reinterpret_cast<char*>(data.data()), w->length);

    // Wrap the raw pointer in a LiteRT tensor buffer.
    LiteRtTensorBufferPtr buf;
    litert::internal::MakeTensorBufferFromData(data.data(), w->length, &buf);

    // Tell the loader the host buffer is ready.
    weight_loader::WeightAccess access;
    access.SetHostBuffer(std::move(buf));
    loader->SetExternalWeightByBuffer(info.external_buffer_id, std::move(access));
  }

  // 5️⃣ Pass the populated loader to the LiteRT compiled model.
  litert::CompiledModel compiled;
  compiled.SetWeightLoader(std::move(loader));
  // Now you can execute the model; the runtime will read the external weights.
}

```

The loader utilizes `mmap` on non-Windows platforms for zero-copy file mapping, as defined in the `CpuMapping` struct within `external_weight_loader_litert.cc`. Each weight file contains raw bytes corresponding to a specific external buffer ID tracked in the `LiteRtWeightInfo` structure (lines 81–98 of the implementation).

## Combining Quantization with External Weights

For production deployments of large language models or vision transformers, combine both techniques to achieve maximum size reduction:

1. **Quantize** the model first using `--post_training_quantize` to reduce weight precision to INT8.
2. **Export external weights** during conversion to separate the now-quantized tensors into individual files.
3. **Package** the tiny metadata flatbuffer with the weight files for distribution.
4. **Load** via the C++ `WeightLoader` API at runtime, enabling LiteRT to execute with quantized hybrid kernels while streaming weights from disk or network storage.

This pipeline reduces both the per-operator memory footprint and the initial download size, making multi-gigabyte models feasible on mobile and IoT devices.

## Summary

- **Post-training weight quantization** reduces model size by approximately 75% by converting float32 weights to INT8, implemented in `tflite/tools/optimize/quantize_model.cc`.
- The **hybrid kernel** execution mode maintains float32 activations for numerical stability while benefiting from compressed weights.
- **External weight files** bypass the 2 GB flatbuffer limit by storing weights separately, controlled via [`weight_loader/external_weight_loader_litert.h`](https://github.com/google-ai-edge/LiteRT/blob/main/weight_loader/external_weight_loader_litert.h).
- On supported platforms, the external weight loader uses **memory mapping** (`mmap`) to avoid loading entire weight files into RAM.
- Both techniques can be combined to optimize model size with LiteRT for large-scale edge deployments.

## Frequently Asked Questions

### What is the difference between weight quantization and full integer quantization in LiteRT?

**Weight quantization** (post-training) only converts the model's weight tensors to INT8 while keeping activations in float32, allowing the model to run on hybrid kernels without retraining. Full integer quantization also converts activations to INT8 and requires representative datasets for calibration. Weight quantization is simpler to implement and suitable for most size-reduction needs, as detailed in [`tflite/tools/optimize/g3doc/quantize_weights.md`](https://github.com/google-ai-edge/LiteRT/blob/main/tflite/tools/optimize/g3doc/quantize_weights.md).

### How do I handle models larger than 2 GB in LiteRT?

Use the **external weight file** mechanism. During conversion, specify the external file path to write weight tensors to separate binary files. At runtime, use `CreateLiteRtWeightLoader` from [`weight_loader/external_weight_loader_litert.h`](https://github.com/google-ai-edge/LiteRT/blob/main/weight_loader/external_weight_loader_litert.h) to map these files into memory. This keeps the `.tflite` flatbuffer under the size limit while the loader handles multi-gigabyte weight payloads via file mapping.

### Can I use quantization and external weights together?

Yes. First apply `--post_training_quantize` to reduce the data size of each weight tensor to one byte per value, then export these quantized weights to external files. This combination minimizes both the metadata flatbuffer size and the external file footprint, achieving the smallest possible distribution package for edge deployment.

### What accuracy loss should I expect from post-training weight quantization?

Most models experience a modest accuracy drop of 1–3% when converting from float32 to INT8 weights, depending on the layer types and weight distributions. The `QuantizeWeights` function in `quantize_model.cc` automatically determines optimal quantization parameters per tensor to minimize this loss. For accuracy-critical applications, consider quantization-aware training rather than post-training quantization.