How to Get Started with LiteRT: A Complete Guide to On-Device AI Inference

LiteRT is Google's open-source framework for high-performance machine learning and generative AI inference on edge devices, providing a compiler, runtime, and vendor-specific acceleration plugins that execute TensorFlow Lite models efficiently across CPUs, GPUs, and NPUs.

LiteRT (the evolution of TensorFlow Lite) is Google’s next-generation runtime for deploying AI models directly on mobile, embedded, and IoT devices without cloud dependencies. This guide walks you through the essential first steps to build and run inference using the google-ai-edge/LiteRT repository, covering everything from the core architecture to practical Python and C++ implementations.

Understanding the LiteRT Architecture

Before writing code, it helps to understand how the runtime is organized. LiteRT separates concerns into distinct layers that handle model compilation, memory management, and hardware dispatch.

Core Components

The repository is organized into four primary areas:

• Core runtime – Located in litert/runtime/, this layer manages model execution, memory pools, and hardware dispatch. It implements the Accelerator registry, TensorBuffer memory abstraction, CompiledModel execution context, and Event synchronization primitives.
• C/C++ public APIs – Stable ABI wrappers found in litert/c/ that expose functions like LiteRtTensorBufferT and LiteRtCompiledModel for application developers.
• Compiler and conversion tools – Found under tflite/ and litert/vendors/, these tools convert TensorFlow and PyTorch models into optimized .tflite format for target hardware.
• Vendor plugins – Hardware-specific implementations under litert/vendors/ for Qualcomm, MediaTek, Intel OpenVINO, and other accelerators.

Key Abstractions

As implemented in the runtime headers (accelerator.h, tensor_buffer.h, compiled_model.h), four concepts drive