Main Features of LiteRT: On-Device AI Runtime Capabilities

LiteRT provides a unified, cross-platform runtime for high-performance on-device machine learning, featuring advanced GPU/NPU acceleration, a compiled model API with async execution, dynamic tensor resizing, and specialized optimizations for generative AI workloads.

LiteRT is Google-AI-Edge’s production-ready runtime for deploying machine learning and generative AI models directly on edge devices. As the successor to TensorFlow Lite, it abstracts hardware complexity through a unified API while delivering optimized performance across Android, iOS, web, and embedded platforms. Understanding the main features of LiteRT helps developers leverage hardware accelerators and advanced execution patterns for everything from computer vision to large language model inference.

Hardware Acceleration with Unified NPU Support

LiteRT delivers high-performance inference by leveraging diverse hardware accelerators through a single, vendor-agnostic interface. The runtime automatically utilizes OpenGL and OpenCL on Android, Metal on iOS and macOS, and WebGPU for browser-based deployment.

For specialized neural processing units, LiteRT provides unified NPU acceleration that eliminates the need for vendor-specific code. According to the source code in litert/vendors/, the runtime includes dedicated delegates for Qualcomm QNN, Samsung NPU, MediaTek NeuroPilot, and Intel OpenVINO. This architecture allows models to target heterogeneous silicon without source code changes, automatically selecting the optimal execution path based on available hardware.

CompiledModel API for Streamlined Execution

At the core of LiteRT is the CompiledModel API, defined in litert/cc/litert_compiled_model.h, which encapsulates the complete lifecycle of model loading, compilation, and inference. This end-to-end workflow automatically selects the optimal accelerator based on device capabilities and model requirements.

The API supports asynchronous execution, enabling non-blocking inference that overlaps compute work with host-side processing. Developers enable this mode through litert::Options (defined in litert/cc/litert_options.h), allowing for reduced latency in production pipelines where CPU and accelerator work can proceed in parallel.

#include "litert/cc/litert_compiled_model.h"
#include "litert/cc/litert_environment.h"
#include "litert/cc/litert_tensor_buffer.h"

int main() {
  // 1️⃣ Create an environment (runtime & delegates are auto‑selected)
  auto env = litert::Environment::Create();

  // 2️⃣ Build default compilation options (auto‑select accelerator)
  auto options = litert::Options::Create().value();

  // 3️⃣ Load and compile a .tflite model file
  auto compiled = litert::CompiledModel::Create(*env, "model.tflite", options)
                      .value();

  // 4️⃣ Query buffer requirements for signature 0 (default)
  auto input_reqs  = compiled.InputRequirements(0);
  auto output_reqs = compiled.OutputRequirements(0);

  // 5️⃣ Allocate TensorBuffers based on the requirements
  auto inputs  = compiled.CreateInputBuffers(0).value();
  auto outputs = compiled.CreateOutputBuffers(0).value();

  // 6️⃣ Fill input buffers (example: copy raw image data)
  std::memcpy(inputs[0].data(), raw_image.data(), inputs[0].size());

  // 7️⃣ Run inference (synchronous; async can be enabled via Options)
  compiled.Invoke(/*signature_index=*/0, inputs, outputs).value();

  // 8️⃣ Process output (e.g., read classification scores)
  const float* scores = reinterpret_cast<const float*>(outputs[0].data());
}

Dynamic Input Tensor Resizing

Unlike static inference runtimes, LiteRT supports dynamic input-tensor resizing at runtime, essential for models handling variable input dimensions such as object detectors with different resolutions. The ResizeInputTensor method, documented in g3doc/apis/litert_resize_api.md, allows shape modification after compilation without rebuilding the model.

// Assume `compiled` is a CompiledModel already created.
std::array<int, 4> new_shape = {1, 720, 1280, 3};   // batch, H, W, C
LITERT_ASSERT_OK(compiled.ResizeInputTensor(/*signature_index=*/0,
                                            /*input_index=*/0,
                                            absl::MakeSpan(new_shape)));

// Re‑query requirements after resize
auto inputs = compiled.CreateInputBuffers(/*signature_index=*/0).value();

Generative AI and LLM Optimizations

LiteRT includes specialized kernels and memory management strategies optimized for generative AI workloads, including large language models and diffusion models. These optimizations focus on efficient attention mechanisms and KV-cache management to deliver what the project describes as "the simplest integration with the best performance" for on-device generative applications.

Cross-Platform Deployment and Language Bindings

The runtime supports comprehensive platform coverage including Android, iOS, Linux, macOS, Windows, Web (via WebGPU), and IoT devices. LiteRT exposes functionality through both a native C++ API for production systems and a thin Python wrapper for rapid prototyping.

The Python bindings, implemented in litert/python/litert_wrapper/compiled_model_wrapper/compiled_model_wrapper.h, mirror the C++ CompiledModel workflow while providing numpy-compatible tensor buffers.

from ai_edge_litert.compiled_model import CompiledModel
from ai_edge_litert.environment import Environment

env = Environment()
model = CompiledModel.from_file("model.tflite", env=env)

# Allocate buffers (the wrapper returns numpy views)

inputs = model.create_input_buffers()
outputs = model.create_output_buffers()

# Fill input, run, and read output

inputs[0][:] = image_array  # shape must match model input

model.invoke()
result = outputs[0]

Runtime Configuration and Built-In Profiling

Fine-grained control over execution behavior is available through the extensible options system centered on litert::Options in litert/cc/litert_options.h. This configuration object manages compiler flags, delegate selection, and hardware-specific knobs.

For performance analysis, LiteRT provides built-in profiling capabilities via litert/cc/litert_profiler.h, enabling per-operation latency measurements and memory usage tracking without external tools.

Summary

• LiteRT unifies GPU, NPU, and CPU execution behind a single vendor-agnostic API, supporting OpenGL, Metal, WebGPU, and vendor-specific delegates in litert/vendors/.
• The CompiledModel API in litert_compiled_model.h provides end-to-end model management with optional asynchronous execution for reduced latency.
• Dynamic tensor resizing via ResizeInputTensor allows runtime shape changes for variable-dimension models without recompilation.
• Specialized generative AI optimizations support efficient LLM and diffusion model inference on resource-constrained devices.
• Cross-platform C++ and Python bindings enable deployment across mobile, desktop, web, and IoT environments.
• Built-in profiling and extensible options provide production-ready observability and hardware tuning capabilities.

Frequently Asked Questions

What hardware accelerators does LiteRT support?

LiteRT supports GPU acceleration via OpenGL/OpenCL on Android, Metal on iOS/macOS, and WebGPU for web browsers. For NPU acceleration, it provides unified delegates for Qualcomm, Samsung, MediaTek, and Intel hardware, accessible through the same API without vendor-specific code. The runtime automatically selects the best available accelerator during model compilation.

How does LiteRT handle dynamic input shapes?

LiteRT provides the ResizeInputTensor method in the CompiledModel API, allowing developers to change input dimensions at runtime. This feature is documented in g3doc/apis/litert_resize_api.md and enables support for variable-resolution models like object detectors without requiring model recompilation or memory reallocation.

What is the difference between LiteRT and TensorFlow Lite?

LiteRT is the next-generation runtime from Google-AI-Edge that succeeds TensorFlow Lite, offering a unified NPU API, improved generative AI optimizations, and a streamlined CompiledModel interface. While maintaining compatibility with TFLite models, LiteRT provides enhanced hardware abstraction and async execution capabilities not available in the legacy runtime.

Can I use LiteRT with Python for prototyping?

Yes, LiteRT provides a thin Python wrapper that mirrors the C++ CompiledModel API, implemented in litert/python/litert_wrapper/compiled_model_wrapper/compiled_model_wrapper.h. This wrapper supports numpy array inputs and provides the same automatic accelerator selection and dynamic resizing features as the native C++ interface.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}