What Is LiteRT For: Google's On-Device ML and GenAI Inference Framework

LiteRT is Google's high-performance runtime framework for deploying machine learning and generative AI models directly on edge devices, offering a unified C++ API that abstracts hardware accelerators across Android, iOS, Linux, macOS, Windows, and Web platforms.

LiteRT, housed in the google-ai-edge/LiteRT repository, serves as the evolutionary successor to TensorFlow Lite, streamlining on-device inference through a modernized architecture. The framework converts standard .tflite models into hardware-optimized internal representations and executes them via a simple CompiledModel API that automatically selects the best available accelerator for the target platform.

Core Architecture and Components

The CompiledModel Abstraction

The heart of LiteRT lives in litert/cc/litert_compiled_model.h, where the litert::CompiledModel class encapsulates the compiled model graph and execution context. This class provides the primary interface for model lifecycle management, offering methods such as GetInputBufferRequirements() and GetOutputBufferRequirements() to inspect tensor specifications, and Run() / RunAsync() to trigger inference. According to the source code, the CompiledModel constructor and factory methods handle the transition from flatbuffer representation to hardware-specific compiled binaries.

Hardware Abstraction and Options

Hardware accelerator selection is configured through the litert::Options class defined in litert/cc/litert_options.h. Developers specify preferred backends using bitwise flags like litert::HwAccelerators::kCpu, kGpu, kNpu, or kWebGpu, allowing the runtime to dispatch operations to CPU, GPU, Neural Processing Unit, or WebGPU backends. The framework automatically loads vendor-specific delegate implementations—such as those found in litert/vendors/qualcomm/dispatch/BUILD—when NPU acceleration is requested.

Runtime Execution Model

Environment Setup and Resource Management

Every LiteRT application begins by instantiating litert::Environment, declared in litert/cc/litert_environment.h, which initializes the runtime state and manages device-level resources. This environment object is passed to litert::CompiledModel::Create() along with the model file path and hardware options, establishing the execution context. The compilation process transforms the .tflite model into an internal representation optimized for the specified accelerators.

Tensor Buffer Management

LiteRT decouples memory management from execution through the litert::TensorBuffer class. The workflow involves creating input and output buffers via CreateInputBuffers() and CreateOutputBuffers(), populating them using type-safe Write<T>() methods, executing inference, and retrieving results with Read<T>(). This architecture supports dynamic tensor shapes that vary between inference calls and enables zero-copy data transfers when hardware backends allow direct memory access.

Key Capabilities and Platform Support

• Cross-Platform Deployment: A single C++ codebase targets Android, iOS, Linux, macOS, Windows, and WebAssembly environments through hardware-agnostic abstractions.

• Advanced Hardware Acceleration: Automatic selection of optimal backends with manual override capabilities via SetHardwareAccelerators(), including support for Qualcomm and MediaTek NPUs through vendor dispatch libraries.

• Asynchronous Execution: Non-blocking inference via RunAsync() with integrated cancellation callbacks, critical for maintaining UI responsiveness during long-running generative AI operations.

• Dynamic Shape Support: Runtime adjustment of input tensor dimensions without requiring model recompilation or regeneration.

• Error Reporting: Comprehensive status return patterns and error propagation through the C++ wrapper API.

Implementation Example

The following C++ example demonstrates the complete seven-step LiteRT workflow, adapted from cmake_example/run_model_simple.cc:

#include "litert/cc/litert_environment.h"
#include "litert/cc/litert_options.h"
#include "litert/cc/litert_compiled_model.h"
#include "litert/cc/litert_tensor_buffer.h"

int main(int argc, char** argv) {
  // 1️⃣ Create an environment (holds runtime & device resources).
  auto env_res = litert::Environment::Create({});
  if (!env_res) return 1;
  litert::Environment env = std::move(*env_res);

  // 2️⃣ Choose the accelerator(s) you want to use.
  litert::Options opts = *litert::Options::Create();
  opts.SetHardwareAccelerators(litert::HwAccelerators::kCpu |
                               litert::HwAccelerators::kGpu);

  // 3️⃣ Load and compile the model file.
  auto model_res = litert::CompiledModel::Create(
      env, "my_model.tflite", opts);
  if (!model_res) return 1;
  litert::CompiledModel model = std::move(*model_res);

  // 4️⃣ Create input & output buffers for the default signature (index 0).
  auto inputs_res  = model.CreateInputBuffers();
  auto outputs_res = model.CreateOutputBuffers();
  if (!inputs_res || !outputs_res) return 1;

  // 5️⃣ (Optional) Fill inputs – here we just zero‑initialize.
  for (auto& buf : *inputs_res) {
    buf.Write<float>(absl::MakeConstSpan(std::vector<float>(buf.Size(), 0.f)));
  }

  // 6️⃣ Run inference synchronously.
  if (!model.Run(*inputs_res, *outputs_res).ok()) return 1;

  // 7️⃣ Read results.
  for (const auto& out : *outputs_res) {
    std::vector<float> data(out.Size());
    out.Read<float>(absl::MakeSpan(data));
    // Process `data` …
  }
  return 0;
}

To restrict execution to NPU hardware only, modify the options object before model creation:

litert::Options opts = *litert::Options::Create();
opts.SetHardwareAccelerators(litert::HwAccelerators::kNpu);  // NPU only
auto model = litert::CompiledModel::Create(env, "npu_model.tflite", opts);

Summary

• LiteRT is Google's unified runtime for on-device ML and GenAI inference, replacing TensorFlow Lite with a modernized, hardware-agnostic C++ API.
• The CompiledModel class in litert/cc/litert_compiled_model.h provides the core abstraction for model loading, buffer management, and both synchronous and asynchronous execution.
• Hardware acceleration is configured through litert::Options in litert/cc/litert_options.h, supporting CPU, GPU, NPU, and WebGPU backends with automatic vendor delegate selection.
• The framework supports dynamic tensor shapes, cancellation callbacks, and cross-platform deployment across mobile, desktop, and web environments.
• Implementation requires three essential components: an Environment for resource management, Options for hardware selection, and a CompiledModel for inference execution.

Frequently Asked Questions

What is the difference between LiteRT and TensorFlow Lite?

LiteRT is the evolutionary successor to TensorFlow Lite, built on top of the TensorFlow Lite C API but providing a higher-level, hardware-agnostic C++ interface. While TensorFlow Lite required manual delegate configuration for hardware acceleration, LiteRT abstracts GPU, NPU, and WebGPU selection through the litert::Options class, significantly simplifying cross-platform deployment and maintenance.

Which hardware accelerators does LiteRT support?

LiteRT supports CPU, GPU (via OpenGL and Metal delegates), NPU (Neural Processing Unit), and WebGPU backends. The specific accelerators available depend on the target platform and vendor implementations located in litert/vendors/*/dispatch/. Developers control accelerator selection programmatically using the SetHardwareAccelerators() method defined in litert/cc/litert_options.h.

Can LiteRT run models converted from PyTorch?

Yes, LiteRT executes models converted to the standard .tflite format, including those originally trained in PyTorch and converted through appropriate export tooling. The litert::CompiledModel::Create() method accepts .tflite files regardless of the original training framework, compiling them into optimized internal representations for the target hardware.

Does LiteRT support asynchronous inference for real-time applications?

Yes, LiteRT provides RunAsync() methods within the CompiledModel class for non-blocking inference execution. This capability is essential for real-time generative AI applications where UI responsiveness must be maintained, and includes support for cancellation callbacks to abort long-running operations before completion.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}