What Types of AI Models Can LiteRT Run? A Complete Guide to On-Device AI

LiteRT can run any TensorFlow Lite (.tflite) model, including vision classifiers, natural language processors, audio detectors, and generative AI models like LLMs and diffusion models, while automatically selecting the optimal CPU, GPU, or NPU accelerator.

The google-ai-edge/LiteRT repository provides a high-performance runtime for deploying AI models directly on edge devices. Whether you need to run image classification, sentiment analysis, or large language model inference, LiteRT executes these workloads efficiently by leveraging hardware-specific delegates. This guide explains exactly what types of AI models LiteRT supports and how it processes them at the source code level.

Supported AI Model Categories in LiteRT

Vision Models

LiteRT provides first-class APIs for computer vision tasks through the Task Library. As documented in tflite/g3doc/inference_with_metadata/task_library/overview.md, supported vision APIs include ImageClassifier, ObjectDetector, ImageSegmenter, ImageEmbedder, and ImageSearcher. These APIs handle models ranging from MobileNet-based classifiers to complex segmentation networks, enabling use cases like object detection, visual search, and feature extraction.

Natural Language Models

For text processing, LiteRT supports NLClassifier, BertNLClassifier, BertQuestionAnswerer, TextSearcher, and TextEmbedder. These task-specific wrappers handle sentiment analysis, intent classification, question answering, and semantic search models. The Bert-based implementations allow running transformer models optimized for on-device constraints.

Audio Models

LiteRT supports audio classification through the AudioClassifier API, enabling keyword spotting and sound event detection models. This covers models trained to identify specific audio patterns or speech commands in real-time on device.

Generative AI Models

Through LiteRT-LM, the runtime supports large language models (LLMs) and diffusion models for on-device text generation and image generation. This extends LiteRT beyond traditional discriminative models into generative AI workloads, allowing models like Gemma 2B to run locally.

Custom and Flex Delegate Models

LiteRT can execute any .tflite model, including those with custom TensorFlow operations, via the Flex delegate. This allows models using unsupported ops to run on-device by falling back to TensorFlow kernels when native TFLite kernels aren't available.

How LiteRT Executes Different Model Types

The execution pipeline in LiteRT follows a structured dispatch system that adapts to model requirements.

Model Loading: The runtime parses the flat-buffer format using FlatBufferModel and constructs a graph of TfLiteNode objects. This occurs in the core compilation phase handled by litert/cc/litert_compiled_model.h.

Operator Dispatch: Each node routes to either a built-in kernel, a hardware delegate (GPU, NPU), or the Flex delegate. The decision logic resides in functions like IsNodeSupportedByDelegate, with platform-specific implementations found in files like tflite/delegates/coreml/README.md for Core ML support.

Execution: The CompiledModel::Run method executes the graph on the selected backend. According to cmake_example/run_model_simple.cc, this process handles tensor allocation through TensorBuffer abstractions that support zero-copy GPU buffers when available.

Task-Specific Helpers: High-level wrappers in C++, Java, and Swift (such as ImageClassifier) abstract the low-level details. These helpers, documented in the Task Library overview, provide simple inference flows while internally managing the complex delegate selection.

Code Examples for Running AI Models in LiteRT

Running a Generic .tflite Model (C++)

The simplest way to run any TensorFlow Lite model uses the minimal example in cmake_example/run_model_simple.cc:

// Compile and run any .tflite model, choosing hardware automatically.
int main(int argc, char** argv) {
  absl::ParseCommandLine(argc, argv);
  // Pass the model path with --graph=your_model.tflite
  // Choose accelerators with --accelerator=cpu,gpu,npu
  auto status = litert::RunModel();
  return status ? EXIT_SUCCESS : EXIT_FAILURE;
}

Vision Task Library Implementation (C++)

For structured vision tasks, use the Task Library API:

// High-level API: only a few lines needed.
auto options = ImageClassifierOptions::Create();
options->SetModelFile("mobilenet_v2.tflite");
auto classifier = ImageClassifier::Create(options).value();

ImageData image = LoadImage("cat.jpg");
auto result = classifier->Classify(image).value();
for (const auto& label : result.classifications()) {
  std::cout << label.id << ": " << label.score << "\n";
}

Large Language Model Inference (Python)

LiteRT-LM provides Python bindings for generative models:

# LiteRT-LM provides a simple Python wrapper.

from litert_lm import LLM
model = LLM("gemma_2b.tflite")
output = model.generate("Explain quantum computing in one sentence.")
print(output)

Key Source Files for Model Support

Understanding the repository structure helps clarify LiteRT's model handling capabilities:

• litert/cc/litert_compiled_model.h: Defines the core CompiledModel class for loading .tflite files and preparing them for execution.
• litert/cc/litert_environment.h: Manages runtime resources including thread pools and hardware backend initialization.
• litert/cc/litert_options.h: Configures hardware accelerators, memory budgets, and execution flags for different model types.
• tflite/g3doc/inference_with_metadata/task_library/overview.md: Documents all supported task-specific APIs and their corresponding model families.
• cmake_example/run_model_simple.cc: Demonstrates minimal end-to-end execution of any .tflite model with automatic accelerator selection.
• tflite/delegates/coreml/README.md: Illustrates how LiteRT maps operators to platform-specific delegates like Core ML, GPU, and NPU.

Summary

• LiteRT executes any TensorFlow Lite (.tflite) model, including quantized, float, and control-flow variants.
• First-class APIs support vision (classification, detection, segmentation), NLP (BERT, text classification), and audio (keyword spotting) tasks.
• LiteRT-LM enables on-device execution of generative AI models including large language models and diffusion models.
• The Flex delegate extends support to custom TensorFlow operations not natively implemented in TFLite.
• Hardware acceleration is automatic, with delegates for CPU, GPU (OpenCL/OpenGL), and vendor-specific NPUs (Qualcomm, MediaTek).

Frequently Asked Questions

Can LiteRT run PyTorch or ONNX models directly?

No, LiteRT specifically executes TensorFlow Lite (.tflite) models. To run PyTorch or ONNX models, you must first convert them to the TFLite format using conversion tools like ONNX-TFLite or PyTorch's mobile exporter, then load the resulting .tflite file via litert/cc/litert_compiled_model.h.

Does LiteRT support quantized models?

Yes, LiteRT fully supports quantized models including INT8 and post-training quantization. The CompiledModel API in litert_compiled_model.h handles quantized tensors automatically, and hardware delegates like GPU and NPU often provide optimized paths for quantized inference.

How do I run a custom TensorFlow operation with LiteRT?

Use the Flex delegate by including the custom op in your model and ensuring the Flex delegate is linked. The runtime checks IsNodeSupportedByDelegate to determine if an op requires Flex fallback, allowing unsupported ops to execute via TensorFlow kernels on-device.

What hardware accelerators does LiteRT support?

LiteRT automatically selects from CPU, GPU (via OpenCL or OpenGL), and NPU backends including Qualcomm QNN and MediaTek Neuron. The selection logic in cmake_example/run_model_simple.cc and related delegate files evaluates the model's operator set against each accelerator's capabilities to optimize performance.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}