LiteRT Usage Examples: Complete Guide to On-Device Inference

You can find official LiteRT usage examples in the cmake_example/ directory of the google-ai-edge/LiteRT repository, including minimal interpreter demos and full CompiledModel API implementations that demonstrate hardware accelerator selection and end-to-end inference.

The LiteRT runtime (formerly TensorFlow Lite) provides self-contained examples in the google-ai-edge/LiteRT repository that demonstrate everything from basic model loading to advanced hardware-accelerated inference. These examples serve as the definitive reference for implementing on-device machine learning in C++ applications.

Where to Find LiteRT Usage Examples

The repository organizes working examples into three main categories: minimal interpreter demonstrations, full runtime API implementations, and vendor extension samples.

Minimal TensorFlow Lite Interpreter

Located at cmake_example/tflite_minimal.cc, this bare-bones program demonstrates direct TensorFlow Lite API usage without LiteRT abstractions. It loads a .tflite file, builds an interpreter using tflite::InterpreterBuilder, allocates tensors, and runs inference.

Full LiteRT CompiledModel API

The cmake_example/run_model_simple.cc file provides a complete end-to-end example using the modern LiteRT runtime. It creates a litert::Environment, selects hardware accelerators (CPU/GPU/NPU), builds a litert::CompiledModel, manages input/output buffers, and executes the model.

Vendor Extension Examples

For custom backend development, litert/vendors/examples/example_transformations.cc shows how to plug custom transformation passes into the LiteRT optimization pipeline. Python developers can reference litert/python/aot/vendors/example/example_backend.py for Ahead-of-Time (AOT) compiler integration.

Key LiteRT APIs Demonstrated

The examples illustrate the core LiteRT runtime workflow through specific API calls implemented in the litert/cc/ directory.

Environment Creation

The litert::Environment::Create method initializes global runtime state, loading system libraries and preparing hardware interfaces. This is implemented in litert/cc/litert_environment.h and litert/cc/litert_environment.cc.

Hardware Accelerator Selection

The Options class manages hardware acceleration through SetHardwareAccelerators. The example parses command-line flags to configure CPU, GPU, or NPU backends via the HwAcceleratorSet structure, defined in litert/cc/litert_options.h.

Model Compilation and Execution

litert::CompiledModel::Create compiles FlatBuffer models for selected accelerators. The Run method executes inference using prepared TensorBuffer objects for inputs and outputs, as defined in litert/cc/litert_compiled_model.h. The CreateInputBuffers and CreateOutputBuffers methods allocate type-safe memory buffers matching the model signature.

Complete Code Examples

Minimal TFLite Interpreter

This example from cmake_example/tflite_minimal.cc shows the low-level TensorFlow Lite approach:

int main(int argc, char* argv[]) {
  if (argc != 2) {
    fprintf(stderr, "tflite_minimal <tflite model>\n");
    return 1;
  }
  const char* filename = argv[1];

  // Load the model.
  std::unique_ptr<tflite::FlatBufferModel> model =
      tflite::FlatBufferModel::BuildFromFile(filename);
  TFLITE_MINIMAL_CHECK(model != nullptr);

  // Build the interpreter.
  tflite::ops::builtin::BuiltinOpResolver resolver;
  tflite::InterpreterBuilder builder(*model, resolver);
  std::unique_ptr<tflite::Interpreter> interpreter;
  builder(&interpreter);
  TFLITE_MINIMAL_CHECK(interpreter != nullptr);

  // Allocate tensors, run, and print state.
  TFLITE_MINIMAL_CHECK(interpreter->AllocateTensors() == kTfLiteOk);
  printf("=== Pre-invoke Interpreter State ===\n");
  tflite::PrintInterpreterState(interpreter.get());

  // TODO: Fill input tensors here.

  TFLITE_MINIMAL_CHECK(interpreter->Invoke() == kTfLiteOk);
  printf("\n=== Post-invoke Interpreter State ===\n");
  tflite::PrintInterpreterState(interpreter.get());

  // TODO: Read output tensors here.
  return 0;
}

LiteRT CompiledModel API

This example from cmake_example/run_model_simple.cc demonstrates the modern LiteRT runtime:

int main(int argc, char** argv) {
  absl::ParseCommandLine(argc, argv);

  // 1️⃣ Create runtime environment.
  LITERT_ASSIGN_OR_RETURN(auto env, litert::Environment::Create({}));

  // 2️⃣ Choose hardware (cpu,gpu,npu) via command-line flag.
  LITERT_ASSIGN_OR_RETURN(auto options, litert::Options::Create());
  options.SetHardwareAccelerators(litert::GetAccelerator());

  // 3️⃣ Load and compile the model.
  LITERT_ASSIGN_OR_RETURN(
      auto compiled_model,
      litert::CompiledModel::Create(env,
                                  absl::GetFlag(FLAGS_graph),  // path to .tflite
                                  options));

  // 4️⃣ Prepare buffers for the specified signature.
  size_t sig = absl::GetFlag(FLAGS_signature_index);
  LITERT_ASSIGN_OR_RETURN(auto input_buffers,
                          compiled_model.CreateInputBuffers(sig));
  // Optionally fill with dummy data for numerical checks.
  for (auto& buf : input_buffers) { LITERT_RETURN_IF_ERROR(FillInputBuffer(buf)); }

  LITERT_ASSIGN_OR_RETURN(auto output_buffers,
                          compiled_model.CreateOutputBuffers(sig));

  // 5️⃣ Run inference.
  litert::Expected<void> status =
      compiled_model.Run(sig, input_buffers, output_buffers);
  if (!status) { ABSL_LOG(ERROR) << status.Error().Message(); return EXIT_FAILURE; }

  ABSL_LOG(INFO) << "Model run completed";
  return EXIT_SUCCESS;
}

Essential Source Files for LiteRT Development

When building applications based on these examples, reference these core header files:

• litert/cc/litert_environment.h – Environment creation and global runtime state management.
• litert/cc/litert_options.h – Configuration container including SetHardwareAccelerators.
• litert/cc/litert_compiled_model.h – CompiledModel class for loading, compiling, and running models.
• litert/cc/litert_tensor_buffer.h – TensorBuffer abstraction for type-safe input/output memory handling.
• cmake_example/CMakeLists.txt – Build configuration for compiling the examples.

Summary

• The cmake_example/ directory contains the primary LiteRT usage examples, ranging from minimal TFLite interpreters to full CompiledModel implementations.
• run_model_simple.cc demonstrates the modern LiteRT workflow: Environment → Options → CompiledModel → TensorBuffer → Run.
• tflite_minimal.cc provides a low-level reference using raw TensorFlow Lite APIs without LiteRT abstractions.
• Vendor extensions and custom backends are illustrated in litert/vendors/examples/ and the Python AOT stubs.
• Key headers in litert/cc/ define the APIs used for environment setup, hardware acceleration, and tensor management.

Frequently Asked Questions

What is the difference between the minimal TFLite example and the LiteRT CompiledModel example?

The minimal example in tflite_minimal.cc uses raw TensorFlow Lite APIs (FlatBufferModel, InterpreterBuilder) directly without LiteRT's abstraction layer. The run_model_simple.cc example uses the modern LiteRT runtime with litert::CompiledModel, which provides hardware accelerator selection, better memory management through TensorBuffer, and a more streamlined API for production deployments.

How do I enable GPU or NPU acceleration in LiteRT examples?

Pass the --accelerator flag when running run_model_simple.cc. The example code parses this flag and calls options.SetHardwareAccelerators() with the appropriate HwAcceleratorSet value. This configuration happens in litert/cc/litert_options.h and is applied during CompiledModel::Create.

Are there Python examples available for LiteRT?

Yes, the repository includes Python AOT (Ahead-of-Time) backend examples in litert/python/aot/vendors/example/example_backend.py. This file demonstrates how to implement custom Python backends for the LiteRT compiler pipeline, though the primary runtime examples are in C++.

How do I build and run these LiteRT examples?

Use the provided cmake_example/CMakeLists.txt to build the examples with CMake. The build system compiles both tflite_minimal.cc and run_model_simple.cc. Run the compiled binaries with a .tflite model file as an argument, and optionally specify --accelerator=gpu or --accelerator=npu for the LiteRT example to test hardware acceleration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}