LiteRT Performance Benchmarks: Measuring Latency, Memory, and Throughput

LiteRT provides a dedicated C++ benchmark tool that measures latency, memory footprint, and throughput of TensorFlow Lite models when executed with LiteRT's compiled-model pipeline.

The google-ai-edge/LiteRT repository includes a comprehensive benchmarking suite designed to evaluate inference performance across diverse hardware accelerators. Understanding these performance benchmarks for LiteRT is essential for optimizing model deployment on edge devices and quantifying the impact of CPU, GPU, and NPU backends.

Benchmark Architecture and Design

The LiteRT benchmark harness extends TensorFlow Lite's benchmark_model infrastructure but replaces the standard interpreter with a Litert compiled model. This architecture enables accurate measurement of hardware-accelerator backends and runtime-specific optimizations that standard TFLite benchmarks cannot capture.

In litert/tools/benchmark_litert_model.h, the BenchmarkLiteRtModel class orchestrates the entire workflow, while BenchmarkLoggingListener handles metric formatting and protobuf serialization. The entry point in litert/tools/benchmark_litert_model_main.cc (lines 24-36) instantiates the benchmark class and invokes its Run() method.

Three-Stage Benchmark Execution Flow

The performance benchmarks for LiteRT follow a structured pipeline implemented in litert/tools/benchmark_litert_model.cc, consisting of three distinct phases.

Model Loading and Compilation

The benchmark initializes with BenchmarkLiteRtModel::LoadModel(), which reads a .tflite file or file descriptor and creates a litert::Model (lines 200-206). The CreateCompiledModelOptions() function constructs a litert::Options object based on command-line flags specifying hardware selection (CPU/GPU/NPU), thread count, FP16 precision, and weight sharing (lines 61-84). The compiled model is instantiated via litert::CompiledModelNext::Create() and stored for execution.

Input Preparation

Input tensors are allocated and populated using either random data or user-defined ranges. The PopulateInputValueRanges() function parses the input_layer_value_range flag to establish input boundaries (lines 210-236), while PrepareInputData() writes generated values into the model's input buffers.

Execution and Metric Collection

During the run phase, RunImpl() executes the model repeatedly while litert::Profiler captures timing statistics when profiling is enabled. The BenchmarkLoggingListener::OnBenchmarkEnd() method (lines 78-115 in benchmark_litert_model.h) formats latency statistics, memory usage, and throughput metrics. Results serialize into a BenchmarkResult protobuf defined in tflite/tools/benchmark/proto/benchmark_result.proto (lines 24-73), capturing average, minimum, maximum, and percentile latencies, initialization and overall memory consumption, and model throughput.

Running Performance Benchmarks for LiteRT

Execute the benchmark tool on desktop environments to compare CPU and GPU acceleration:

# Build the binary

bazel build -c opt //litert/tools:benchmark_litert_model_main

# Execute the benchmark

bazel-bin/litert/tools/benchmark_litert_model_main \
  --graph=/path/to/model.tflite \
  --use_cpu=true \
  --use_gpu=true \
  --gpu_backend=opengl \
  --allow_fp16=true \
  --num_threads=4 \
  --num_runs=100 \
  --enable_profiler=true \
  --result_file_path=/tmp/benchmark.pb

Deploy and run on Android devices with NPU delegates:

# Build the Android APK

bazel build -c opt --config=android_arm64 //litert/tools:benchmark_litert_model_android

# Install and run

adb install -r -d -g bazel-bin/litert/tools/benchmark_litert_model_android.apk
adb shell /data/local/tmp/benchmark_litert_model \
  --graph=/data/local/tmp/model.tflite \
  --use_npu=true \
  --dispatch_library_path=/data/local/tmp/dispatch_lib \
  --compiler_plugin_library_path=/data/local/tmp/compiler_plugin \
  --num_threads=2

Parse serialized protobuf results using Python:

from litert.tools.benchmark_result_pb2 import BenchmarkResult

with open("/tmp/benchmark.pb", "rb") as f:
    result = BenchmarkResult()
    result.ParseFromString(f.read())

print("Avg latency (ms):", result.latency_metrics.avg_ms)
print("Peak memory (MB):", result.memory_metrics.peak_mem_mb)
print("Throughput (MB/s):", result.misc_metrics.model_throughput_in_mb_per_sec)

Core Source Files

The performance benchmarks for LiteRT rely on these specific components:

• litert/tools/benchmark_litert_model_main.cc: Minimal main() wrapper that drives the benchmarking process.
• litert/tools/benchmark_litert_model.h: Declares BenchmarkLiteRtModel, BenchmarkLoggingListener, metric formatting utilities, and flag definitions.
• litert/tools/benchmark_litert_model.cc: Implements model loading, environment creation, compilation options, input preparation, and run logic.
• tflite/tools/benchmark/proto/benchmark_result.proto: Defines the protobuf schema for persisting latency, memory, and throughput metrics.
• litert/tools/benchmark_stripped_litert_model.cc: Lightweight variant for stripped builds sharing the same metric collection pipeline.

Summary

• LiteRT benchmarks measure latency, memory footprint, and throughput using a C++ tool built on TensorFlow Lite's infrastructure but specialized for compiled models.
• The three-stage pipeline handles model compilation via CreateCompiledModelOptions(), input preparation through PopulateInputValueRanges(), and execution via RunImpl().
• Hardware backends including CPU, GPU, and NPU are configurable through command-line flags that control the litert::Options object.
• Results serialize into protobuf format with detailed statistics for average, min, max, and percentile latencies stored in BenchmarkResult.
• The tool supports desktop and Android deployments with identical metric collection semantics across platforms.

Frequently Asked Questions

What metrics does the LiteRT benchmark tool capture?

The tool captures latency statistics (average, minimum, maximum, and percentiles), memory metrics (initialization and peak memory usage in MB), and throughput (model throughput in MB/s). These values serialize into a BenchmarkResult protobuf defined in tflite/tools/benchmark/proto/benchmark_result.proto, enabling automated analysis and regression tracking.

How does LiteRT benchmarking differ from standard TensorFlow Lite benchmarks?

LiteRT replaces the standard TFLite interpreter with a Litert compiled model through litert::CompiledModelNext::Create(). This allows the benchmark to evaluate LiteRT-specific optimizations and hardware accelerator delegates (particularly NPU support) that the standard benchmark_model executable cannot measure, providing accurate performance profiles for the LiteRT runtime.

Can I run LiteRT benchmarks on Android devices with NPU acceleration?

Yes. Build the Android target using --config=android_arm64 and specify --use_npu=true along with --dispatch_library_path and --compiler_plugin_library_path pointing to your hardware-specific libraries. The binary collects identical latency and memory metrics on-device as it does on desktop environments, enabling direct comparison of NPU versus CPU performance.

Where are benchmark results stored and how can I parse them?

Results optionally serialize to a file path specified by the --result_file_path flag. The protobuf schema includes fields for latency_metrics.avg_ms, memory_metrics.peak_mem_mb, and misc_metrics.model_throughput_in_mb_per_sec, which you can parse using the Python benchmark_result_pb2 module generated from tflite/tools/benchmark/proto/benchmark_result.proto.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}