# LiteRT Performance Benchmarks: Measuring Latency, Memory, and Throughput

> Discover LiteRT performance benchmarks. Measure latency memory footprint and throughput of TensorFlow Lite models with LiteRTs C++ benchmark tool. Optimize your edge AI performance now.

- Repository: [google-ai-edge/LiteRT](https://github.com/google-ai-edge/LiteRT)
- Tags: performance
- Published: 2026-03-13

---

**LiteRT provides a dedicated C++ benchmark tool that measures latency, memory footprint, and throughput of TensorFlow Lite models when executed with LiteRT's compiled-model pipeline.**

The google-ai-edge/LiteRT repository includes a comprehensive benchmarking suite designed to evaluate inference performance across diverse hardware accelerators. Understanding these performance benchmarks for LiteRT is essential for optimizing model deployment on edge devices and quantifying the impact of CPU, GPU, and NPU backends.

## Benchmark Architecture and Design

The LiteRT benchmark harness extends TensorFlow Lite's `benchmark_model` infrastructure but replaces the standard interpreter with a **Litert compiled model**. This architecture enables accurate measurement of hardware-accelerator backends and runtime-specific optimizations that standard TFLite benchmarks cannot capture.

In [`litert/tools/benchmark_litert_model.h`](https://github.com/google-ai-edge/LiteRT/blob/main/litert/tools/benchmark_litert_model.h), the `BenchmarkLiteRtModel` class orchestrates the entire workflow, while `BenchmarkLoggingListener` handles metric formatting and protobuf serialization. The entry point in `litert/tools/benchmark_litert_model_main.cc` (lines 24-36) instantiates the benchmark class and invokes its `Run()` method.

## Three-Stage Benchmark Execution Flow

The performance benchmarks for LiteRT follow a structured pipeline implemented in `litert/tools/benchmark_litert_model.cc`, consisting of three distinct phases.

### Model Loading and Compilation

The benchmark initializes with `BenchmarkLiteRtModel::LoadModel()`, which reads a `.tflite` file or file descriptor and creates a `litert::Model` (lines 200-206). The `CreateCompiledModelOptions()` function constructs a `litert::Options` object based on command-line flags specifying hardware selection (CPU/GPU/NPU), thread count, FP16 precision, and weight sharing (lines 61-84). The compiled model is instantiated via `litert::CompiledModelNext::Create()` and stored for execution.

### Input Preparation

Input tensors are allocated and populated using either random data or user-defined ranges. The `PopulateInputValueRanges()` function parses the `input_layer_value_range` flag to establish input boundaries (lines 210-236), while `PrepareInputData()` writes generated values into the model's input buffers.

### Execution and Metric Collection

During the run phase, `RunImpl()` executes the model repeatedly while `litert::Profiler` captures timing statistics when profiling is enabled. The `BenchmarkLoggingListener::OnBenchmarkEnd()` method (lines 78-115 in [`benchmark_litert_model.h`](https://github.com/google-ai-edge/LiteRT/blob/main/benchmark_litert_model.h)) formats latency statistics, memory usage, and throughput metrics. Results serialize into a `BenchmarkResult` protobuf defined in `tflite/tools/benchmark/proto/benchmark_result.proto` (lines 24-73), capturing average, minimum, maximum, and percentile latencies, initialization and overall memory consumption, and model throughput.

## Running Performance Benchmarks for LiteRT

Execute the benchmark tool on desktop environments to compare CPU and GPU acceleration:

```bash

# Build the binary

bazel build -c opt //litert/tools:benchmark_litert_model_main

# Execute the benchmark

bazel-bin/litert/tools/benchmark_litert_model_main \
  --graph=/path/to/model.tflite \
  --use_cpu=true \
  --use_gpu=true \
  --gpu_backend=opengl \
  --allow_fp16=true \
  --num_threads=4 \
  --num_runs=100 \
  --enable_profiler=true \
  --result_file_path=/tmp/benchmark.pb

```

Deploy and run on Android devices with NPU delegates:

```bash

# Build the Android APK

bazel build -c opt --config=android_arm64 //litert/tools:benchmark_litert_model_android

# Install and run

adb install -r -d -g bazel-bin/litert/tools/benchmark_litert_model_android.apk
adb shell /data/local/tmp/benchmark_litert_model \
  --graph=/data/local/tmp/model.tflite \
  --use_npu=true \
  --dispatch_library_path=/data/local/tmp/dispatch_lib \
  --compiler_plugin_library_path=/data/local/tmp/compiler_plugin \
  --num_threads=2

```

Parse serialized protobuf results using Python:

```python
from litert.tools.benchmark_result_pb2 import BenchmarkResult

with open("/tmp/benchmark.pb", "rb") as f:
    result = BenchmarkResult()
    result.ParseFromString(f.read())

print("Avg latency (ms):", result.latency_metrics.avg_ms)
print("Peak memory (MB):", result.memory_metrics.peak_mem_mb)
print("Throughput (MB/s):", result.misc_metrics.model_throughput_in_mb_per_sec)

```

## Core Source Files

The performance benchmarks for LiteRT rely on these specific components:

- **`litert/tools/benchmark_litert_model_main.cc`**: Minimal `main()` wrapper that drives the benchmarking process.
- **[`litert/tools/benchmark_litert_model.h`](https://github.com/google-ai-edge/LiteRT/blob/main/litert/tools/benchmark_litert_model.h)**: Declares `BenchmarkLiteRtModel`, `BenchmarkLoggingListener`, metric formatting utilities, and flag definitions.
- **`litert/tools/benchmark_litert_model.cc`**: Implements model loading, environment creation, compilation options, input preparation, and run logic.
- **`tflite/tools/benchmark/proto/benchmark_result.proto`**: Defines the protobuf schema for persisting latency, memory, and throughput metrics.
- **`litert/tools/benchmark_stripped_litert_model.cc`**: Lightweight variant for stripped builds sharing the same metric collection pipeline.

## Summary

- **LiteRT benchmarks** measure latency, memory footprint, and throughput using a C++ tool built on TensorFlow Lite's infrastructure but specialized for compiled models.
- The **three-stage pipeline** handles model compilation via `CreateCompiledModelOptions()`, input preparation through `PopulateInputValueRanges()`, and execution via `RunImpl()`.
- **Hardware backends** including CPU, GPU, and NPU are configurable through command-line flags that control the `litert::Options` object.
- Results serialize into **protobuf format** with detailed statistics for average, min, max, and percentile latencies stored in `BenchmarkResult`.
- The tool supports **desktop and Android deployments** with identical metric collection semantics across platforms.

## Frequently Asked Questions

### What metrics does the LiteRT benchmark tool capture?

The tool captures **latency statistics** (average, minimum, maximum, and percentiles), **memory metrics** (initialization and peak memory usage in MB), and **throughput** (model throughput in MB/s). These values serialize into a `BenchmarkResult` protobuf defined in `tflite/tools/benchmark/proto/benchmark_result.proto`, enabling automated analysis and regression tracking.

### How does LiteRT benchmarking differ from standard TensorFlow Lite benchmarks?

LiteRT replaces the standard TFLite interpreter with a **Litert compiled model** through `litert::CompiledModelNext::Create()`. This allows the benchmark to evaluate LiteRT-specific optimizations and hardware accelerator delegates (particularly NPU support) that the standard `benchmark_model` executable cannot measure, providing accurate performance profiles for the LiteRT runtime.

### Can I run LiteRT benchmarks on Android devices with NPU acceleration?

Yes. Build the Android target using `--config=android_arm64` and specify `--use_npu=true` along with `--dispatch_library_path` and `--compiler_plugin_library_path` pointing to your hardware-specific libraries. The binary collects identical latency and memory metrics on-device as it does on desktop environments, enabling direct comparison of NPU versus CPU performance.

### Where are benchmark results stored and how can I parse them?

Results optionally serialize to a file path specified by the `--result_file_path` flag. The protobuf schema includes fields for `latency_metrics.avg_ms`, `memory_metrics.peak_mem_mb`, and `misc_metrics.model_throughput_in_mb_per_sec`, which you can parse using the Python `benchmark_result_pb2` module generated from `tflite/tools/benchmark/proto/benchmark_result.proto`.