LiteRT Ecosystem Tools: Complete Guide to Command-Line Utilities for Model Deployment

The LiteRT ecosystem includes 13 command-line tools for running, benchmarking, analyzing, and debugging TensorFlow Lite models across CPU, GPU, and NPU hardware, all located in the litert/tools/ directory of the google-ai-edge/LiteRT repository.

The LiteRT (Lite Runtime) project provides a self-contained suite of utilities that enable developers to deploy and validate machine learning models on diverse hardware backends. These tools share a unified architecture based on the ai_edge_litert::CompiledModel C++ API and support hardware-specific optimizations through compiler plugins and dispatch libraries. Whether you need to benchmark latency on an NPU or isolate a numerically unstable operator, the LiteRT ecosystem offers specialized binaries for every stage of the model deployment pipeline.

Model Execution and Benchmarking Tools

The execution tools form the foundation of the LiteRT ecosystem, providing both full-featured and minimal interfaces for model inference.

run_model

run_model is the primary full-featured runner located at litert/tools/run_model.cc. It supports all hardware accelerators, handles input directories, prints tensor values, performs numerical comparisons against CPU baselines, and loads compiler plugins for target-specific optimizations.

The tool parses command-line options through the LitertToolOptions struct defined in litert/tools/flags/flags.h, then initializes a CompiledModel instance. When --accelerator=gpu or --accelerator=npu is specified, the tool automatically loads the appropriate dispatch library from litert/vendors/ using the DispatchLibraryLoader class.

run_model_simple

For quick sanity checks, run_model_simple (litert/tools/run_model_simple.cc) provides a minimal implementation that loads a .tflite file and performs a single inference without additional diagnostics. This tool is ideal for verifying that a model loads correctly on a new hardware target before running comprehensive benchmarks.

benchmark_litert_model_main

Performance analysis relies on benchmark_litert_model_main (litert/tools/benchmark_litert_model_main.cc). This tool executes a model for a configurable number of iterations, reporting detailed latency statistics, throughput metrics, and memory usage. It supports warmup runs to stabilize cache states and can load compiler plugins via the --compiler_plugin_library_dir flag to measure optimized graph performance.

Model Analysis and Inspection Tools

Before deployment, understanding model structure and metadata is critical. The LiteRT ecosystem provides several analysis utilities.

analyze_model_main

analyze_model_main (litert/tools/analyze_model_main.cc) generates high-level model summaries or detailed operator lists, including subgraph structures, tensor shapes, and operation codes. This tool helps developers verify that model conversion preserved the expected architecture.

dump_ops_main

For debugging specific operators, dump_ops_main (litert/tools/dump_ops_main.cc) splits a model into individual-operator sub-models. It supports batch processing and opcode filtering, enabling isolated testing of specific layers that may cause issues on particular hardware.

metadata_util_main

metadata_util_main (litert/tools/metadata_util_main.cc) inspects or extracts model metadata, including custom operator registrations and version information. This is essential when working with models that contain custom operations or specific runtime requirements.

extract_bytecode

The extract_bytecode tool (litert/tools/extract_bytecode.cc) extracts compiled bytecode from LiteRT models for offline inspection. This allows developers to analyze the low-level instructions generated for specific accelerators without executing the model.

Hardware Validation and Debugging Tools

Ensuring numerical correctness across different hardware backends requires specialized validation tools.

npu_numerics_check and gpu_numerics_check

npu_numerics_check (litert/tools/npu_numerics_check.cc) and gpu_numerics_check (litert/tools/gpu_numerics_check.cc) execute models on both CPU and their respective accelerators (NPU or GPU), then compare output tensors numerically. These tools report maximum absolute and relative errors, helping identify precision loss introduced by hardware-specific quantization or operator implementations.

culprit_finder_main

When a model crashes or produces incorrect results, culprit_finder_main (litert/tools/culprit_finder/culprit_finder_main.cc) performs a binary search through the operator list to isolate the specific operation causing the failure. The tool progressively creates sub-models containing smaller slices of the original graph until it identifies the offending operator's index, opcode, and description via the OperatorRegistry.

accuracy_debugger_main

accuracy_debugger_main (litert/tools/accuracy_debugger/accuracy_debugger_main.cc) runs models while logging per-operator accuracy statistics. This granular visibility helps pinpoint where precision loss occurs during dispatch to hardware accelerators, particularly useful when debugging quantized models.

Compiler Plugin and Optimization Tools

LiteRT supports ahead-of-time optimizations through compiler plugins that perform graph transformations for specific SoCs.

apply_plugin_main

apply_plugin_main (litert/tools/apply_plugin_main.cc) applies hardware-specific compiler plugins to a model, producing an optimized output file. The tool loads plugins via PluginLoader (implemented in litert/plugins/plugin_loader.cc) and invokes Plugin::Run to perform target-specific optimizations such as operator fusion or quantization before model execution.

intel_openvino_flags_example

For vendor-specific implementations, intel_openvino_flags_example (litert/tools/flags/vendors/intel_openvino_flags_example.cc) demonstrates how to define and parse backend-specific command-line flags. This serves as a reference for integrating new hardware vendors into the LiteRT ecosystem.

Shared Architecture and Implementation Details

All LiteRT ecosystem tools share common architectural patterns that ensure consistency across the suite.

Unified Flag Parsing

Every tool includes litert/tools/flags/flags.h, which defines the LitertToolOptions struct containing shared command-line options. Flags parse through absl::ParseCommandLine, providing standardized interfaces for --graph, --accelerator, --input_dir, and diagnostic flags like --print_tensors and --compare_numerical.

CompiledModel API Integration

At the core of each tool lies the ai_edge_litert::CompiledModel C++ API. This interface handles .tflite file loading, input/output tensor binding, and inference invocation. The API abstracts hardware differences, allowing the same tool binary to target CPU, GPU, or NPU through configuration flags.

Dispatch Library Loading

When targeting specialized hardware, tools construct a DispatchLibraryLoader that dynamically opens vendor libraries (e.g., libLiteRtDispatch_Qualcomm.so or libLiteRtDispatch_MediaTek.so) using dlopen. The loader resolves the CreateDispatch entry point required by CompiledModel to forward operations to the appropriate NPU or GPU driver.

Tensor I/O and Diagnostics

Input tensors can be supplied as raw binary files in a directory specified by --input_dir. The LoadInputsFromDirectory helper (declared in litert/tools/util.h) maps files to signature input names and performs type-aware casting. When --print_tensors is enabled, PrintTensorValues outputs tensor contents with configurable sample sizes via --sample_size.

Practical Usage Examples

Running a Model on GPU with Tensor Inspection

Build and execute the full-featured runner to validate GPU execution:

bazel build //litert/tools:run_model

./bazel-bin/litert/tools/run_model \
  --graph=/tmp/my_model.tflite \
  --accelerator=gpu \
  --print_tensors=true \
  --sample_size=5

Under the hood, run_model.cc instantiates CompiledModel, selects the GPU dispatch library, runs inference, and calls PrintTensorValues with the specified sample size.

Benchmarking with NPU Acceleration and Compiler Plugins

Measure performance on Qualcomm NPU hardware with graph optimizations:

bazel build //litert/tools:benchmark_litert_model_main

./bazel-bin/litert/tools/benchmark_litert_model_main \
  --graph=/tmp/vision_model.tflite \
  --accelerator=npu \
  --dispatch_library_dir=/opt/qnn_dispatch \
  --compiler_plugin_library_dir=/opt/qnn_plugin \
  --num_runs=100 \
  --warmup_runs=5

The tool loads the NPU dispatch library, pre-loads the compiler plugin via PluginLoader for target-specific optimizations, and reports standardized latency and throughput statistics.

Isolating Crashing Operators

Use the culprit finder to identify unstable operations:

bazel build //litert/tools/culprit_finder:culprit_finder_main

./bazel-bin/litert/tools/culprit_finder/culprit_finder_main \
  --graph=/tmp/faulty_model.tflite \
  --accelerator=npu \
  --dispatch_library_dir=/opt/qnn_dispatch

The binary performs binary search isolation on the operator list, printing the index and opcode of the first operation that triggers a crash or numerical discrepancy.

Summary

• The LiteRT ecosystem provides 13 specialized command-line tools for complete model lifecycle management, from execution to debugging.
• All tools reside in litert/tools/ and share common infrastructure through LitertToolOptions and the CompiledModel API.
• Execution tools like run_model and benchmark_litert_model_main support CPU, GPU, and NPU backends through the dispatch library abstraction.
• Validation tools including npu_numerics_check and culprit_finder_main ensure numerical correctness and isolate hardware-specific failures.
• Compiler plugin tools such as apply_plugin_main enable ahead-of-time optimizations for specific SoC targets.
• The architecture supports modular builds while maintaining consistent CLI patterns across all utilities.

Frequently Asked Questions

What is the difference between run_model and run_model_simple?

run_model is the full-featured runner supporting all accelerators, input directories, tensor printing, numerical comparison, and compiler plugin loading, while run_model_simple provides a minimal implementation that only loads a model and performs a single inference. Use run_model_simple for quick sanity checks when you don't need diagnostic features or hardware acceleration configuration.

How do I validate numerical accuracy between CPU and NPU execution?

Use the npu_numerics_check tool located at litert/tools/npu_numerics_check.cc. This utility runs your model on both CPU and NPU, compares output tensors using max absolute and relative error metrics, and reports discrepancies. For GPU validation, use the equivalent gpu_numerics_check tool instead.

Where are the shared command-line flags defined in the LiteRT ecosystem?

All tools share common flag definitions in litert/tools/flags/flags.h, which declares the LitertToolOptions struct. Each tool parses these flags using absl::ParseCommandLine, ensuring consistent options for model paths, accelerator selection, input directories, and diagnostic settings across the entire tool suite.

How do I apply hardware-specific optimizations to my model before deployment?

Use apply_plugin_main (litert/tools/apply_plugin_main.cc) with the --compiler_plugin_library_dir flag pointing to your vendor-specific plugin. The tool loads the plugin via PluginLoader and invokes Plugin::Run to perform ahead-of-time graph transformations, outputting an optimized model file ready for deployment on the target hardware.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}