# What Is LiteRT For: Google's On-Device ML and GenAI Inference Framework

> Discover LiteRT, Google's high-performance runtime for deploying ML and GenAI on edge devices. Its unified C++ API supports cross-platform hardware acceleration.

- Repository: [google-ai-edge/LiteRT](https://github.com/google-ai-edge/LiteRT)
- Tags: getting-started
- Published: 2026-03-13

---

**LiteRT is Google's high-performance runtime framework for deploying machine learning and generative AI models directly on edge devices, offering a unified C++ API that abstracts hardware accelerators across Android, iOS, Linux, macOS, Windows, and Web platforms.**

LiteRT, housed in the `google-ai-edge/LiteRT` repository, serves as the evolutionary successor to TensorFlow Lite, streamlining on-device inference through a modernized architecture. The framework converts standard `.tflite` models into hardware-optimized internal representations and executes them via a simple `CompiledModel` API that automatically selects the best available accelerator for the target platform.

## Core Architecture and Components

### The CompiledModel Abstraction

The heart of LiteRT lives in [`litert/cc/litert_compiled_model.h`](https://github.com/google-ai-edge/LiteRT/blob/main/litert/cc/litert_compiled_model.h), where the `litert::CompiledModel` class encapsulates the compiled model graph and execution context. This class provides the primary interface for model lifecycle management, offering methods such as `GetInputBufferRequirements()` and `GetOutputBufferRequirements()` to inspect tensor specifications, and `Run()` / `RunAsync()` to trigger inference. According to the source code, the `CompiledModel` constructor and factory methods handle the transition from flatbuffer representation to hardware-specific compiled binaries.

### Hardware Abstraction and Options

Hardware accelerator selection is configured through the `litert::Options` class defined in [`litert/cc/litert_options.h`](https://github.com/google-ai-edge/LiteRT/blob/main/litert/cc/litert_options.h). Developers specify preferred backends using bitwise flags like `litert::HwAccelerators::kCpu`, `kGpu`, `kNpu`, or `kWebGpu`, allowing the runtime to dispatch operations to CPU, GPU, Neural Processing Unit, or WebGPU backends. The framework automatically loads vendor-specific delegate implementations—such as those found in `litert/vendors/qualcomm/dispatch/BUILD`—when NPU acceleration is requested.

## Runtime Execution Model

### Environment Setup and Resource Management

Every LiteRT application begins by instantiating `litert::Environment`, declared in [`litert/cc/litert_environment.h`](https://github.com/google-ai-edge/LiteRT/blob/main/litert/cc/litert_environment.h), which initializes the runtime state and manages device-level resources. This environment object is passed to `litert::CompiledModel::Create()` along with the model file path and hardware options, establishing the execution context. The compilation process transforms the `.tflite` model into an internal representation optimized for the specified accelerators.

### Tensor Buffer Management

LiteRT decouples memory management from execution through the `litert::TensorBuffer` class. The workflow involves creating input and output buffers via `CreateInputBuffers()` and `CreateOutputBuffers()`, populating them using type-safe `Write<T>()` methods, executing inference, and retrieving results with `Read<T>()`. This architecture supports **dynamic tensor shapes** that vary between inference calls and enables zero-copy data transfers when hardware backends allow direct memory access.

## Key Capabilities and Platform Support

- **Cross-Platform Deployment**: A single C++ codebase targets Android, iOS, Linux, macOS, Windows, and WebAssembly environments through hardware-agnostic abstractions.

- **Advanced Hardware Acceleration**: Automatic selection of optimal backends with manual override capabilities via `SetHardwareAccelerators()`, including support for Qualcomm and MediaTek NPUs through vendor dispatch libraries.

- **Asynchronous Execution**: Non-blocking inference via `RunAsync()` with integrated cancellation callbacks, critical for maintaining UI responsiveness during long-running generative AI operations.

- **Dynamic Shape Support**: Runtime adjustment of input tensor dimensions without requiring model recompilation or regeneration.

- **Error Reporting**: Comprehensive status return patterns and error propagation through the C++ wrapper API.

## Implementation Example

The following C++ example demonstrates the complete seven-step LiteRT workflow, adapted from `cmake_example/run_model_simple.cc`:

```cpp
#include "litert/cc/litert_environment.h"
#include "litert/cc/litert_options.h"
#include "litert/cc/litert_compiled_model.h"
#include "litert/cc/litert_tensor_buffer.h"

int main(int argc, char** argv) {
  // 1️⃣ Create an environment (holds runtime & device resources).
  auto env_res = litert::Environment::Create({});
  if (!env_res) return 1;
  litert::Environment env = std::move(*env_res);

  // 2️⃣ Choose the accelerator(s) you want to use.
  litert::Options opts = *litert::Options::Create();
  opts.SetHardwareAccelerators(litert::HwAccelerators::kCpu |
                               litert::HwAccelerators::kGpu);

  // 3️⃣ Load and compile the model file.
  auto model_res = litert::CompiledModel::Create(
      env, "my_model.tflite", opts);
  if (!model_res) return 1;
  litert::CompiledModel model = std::move(*model_res);

  // 4️⃣ Create input & output buffers for the default signature (index 0).
  auto inputs_res  = model.CreateInputBuffers();
  auto outputs_res = model.CreateOutputBuffers();
  if (!inputs_res || !outputs_res) return 1;

  // 5️⃣ (Optional) Fill inputs – here we just zero‑initialize.
  for (auto& buf : *inputs_res) {
    buf.Write<float>(absl::MakeConstSpan(std::vector<float>(buf.Size(), 0.f)));
  }

  // 6️⃣ Run inference synchronously.
  if (!model.Run(*inputs_res, *outputs_res).ok()) return 1;

  // 7️⃣ Read results.
  for (const auto& out : *outputs_res) {
    std::vector<float> data(out.Size());
    out.Read<float>(absl::MakeSpan(data));
    // Process `data` …
  }
  return 0;
}

```

To restrict execution to NPU hardware only, modify the options object before model creation:

```cpp
litert::Options opts = *litert::Options::Create();
opts.SetHardwareAccelerators(litert::HwAccelerators::kNpu);  // NPU only
auto model = litert::CompiledModel::Create(env, "npu_model.tflite", opts);

```

## Summary

- **LiteRT** is Google's unified runtime for on-device ML and GenAI inference, replacing TensorFlow Lite with a modernized, hardware-agnostic C++ API.
- The **CompiledModel** class in [`litert/cc/litert_compiled_model.h`](https://github.com/google-ai-edge/LiteRT/blob/main/litert/cc/litert_compiled_model.h) provides the core abstraction for model loading, buffer management, and both synchronous and asynchronous execution.
- **Hardware acceleration** is configured through `litert::Options` in [`litert/cc/litert_options.h`](https://github.com/google-ai-edge/LiteRT/blob/main/litert/cc/litert_options.h), supporting CPU, GPU, NPU, and WebGPU backends with automatic vendor delegate selection.
- The framework supports **dynamic tensor shapes**, **cancellation callbacks**, and **cross-platform deployment** across mobile, desktop, and web environments.
- Implementation requires three essential components: an `Environment` for resource management, `Options` for hardware selection, and a `CompiledModel` for inference execution.

## Frequently Asked Questions

### What is the difference between LiteRT and TensorFlow Lite?

LiteRT is the evolutionary successor to TensorFlow Lite, built on top of the TensorFlow Lite C API but providing a higher-level, hardware-agnostic C++ interface. While TensorFlow Lite required manual delegate configuration for hardware acceleration, LiteRT abstracts GPU, NPU, and WebGPU selection through the `litert::Options` class, significantly simplifying cross-platform deployment and maintenance.

### Which hardware accelerators does LiteRT support?

LiteRT supports CPU, GPU (via OpenGL and Metal delegates), NPU (Neural Processing Unit), and WebGPU backends. The specific accelerators available depend on the target platform and vendor implementations located in `litert/vendors/*/dispatch/`. Developers control accelerator selection programmatically using the `SetHardwareAccelerators()` method defined in [`litert/cc/litert_options.h`](https://github.com/google-ai-edge/LiteRT/blob/main/litert/cc/litert_options.h).

### Can LiteRT run models converted from PyTorch?

Yes, LiteRT executes models converted to the standard `.tflite` format, including those originally trained in PyTorch and converted through appropriate export tooling. The `litert::CompiledModel::Create()` method accepts `.tflite` files regardless of the original training framework, compiling them into optimized internal representations for the target hardware.

### Does LiteRT support asynchronous inference for real-time applications?

Yes, LiteRT provides `RunAsync()` methods within the `CompiledModel` class for non-blocking inference execution. This capability is essential for real-time generative AI applications where UI responsiveness must be maintained, and includes support for cancellation callbacks to abort long-running operations before completion.