# How to Get Started with LiteRT: A Complete Guide to On-Device AI Inference

> Get started with LiteRT, Google's open-source framework for fast on-device AI inference. Optimize TensorFlow Lite models for CPUs, GPUs, and NPUs with this complete guide.

- Repository: [google-ai-edge/LiteRT](https://github.com/google-ai-edge/LiteRT)
- Tags: getting-started
- Published: 2026-03-13

---

**LiteRT is Google's open-source framework for high-performance machine learning and generative AI inference on edge devices, providing a compiler, runtime, and vendor-specific acceleration plugins that execute TensorFlow Lite models efficiently across CPUs, GPUs, and NPUs.**

LiteRT (the evolution of TensorFlow Lite) is Google’s next-generation runtime for deploying AI models directly on mobile, embedded, and IoT devices without cloud dependencies. This guide walks you through the essential first steps to build and run inference using the `google-ai-edge/LiteRT` repository, covering everything from the core architecture to practical Python and C++ implementations.

## Understanding the LiteRT Architecture

Before writing code, it helps to understand how the runtime is organized. LiteRT separates concerns into distinct layers that handle model compilation, memory management, and hardware dispatch.

### Core Components

The repository is organized into four primary areas:

- **Core runtime** – Located in `litert/runtime/`, this layer manages model execution, memory pools, and hardware dispatch. It implements the **Accelerator** registry, **TensorBuffer** memory abstraction, **CompiledModel** execution context, and **Event** synchronization primitives.
- **C/C++ public APIs** – Stable ABI wrappers found in `litert/c/` that expose functions like `LiteRtTensorBufferT` and `LiteRtCompiledModel` for application developers.
- **Compiler and conversion tools** – Found under `tflite/` and `litert/vendors/`, these tools convert TensorFlow and PyTorch models into optimized `.tflite` format for target hardware.
- **Vendor plugins** – Hardware-specific implementations under `litert/vendors/` for Qualcomm, MediaTek, Intel OpenVINO, and other accelerators.

### Key Abstractions

As implemented in the runtime headers ([`accelerator.h`](https://github.com/google-ai-edge/LiteRT/blob/main/accelerator.h), [`tensor_buffer.h`](https://github.com/google-ai-edge/LiteRT/blob/main/tensor_buffer.h), [`compiled_model.h`](https://github.com/google-ai-edge/LiteRT/blob/main/compiled_model.h)), four concepts drive