# What Is LiteRT and How It Powers On-Device LLM Inference in Google AI Edge Gallery

> Discover LiteRT, the lightweight runtime powering on-device LLM inference with a simple Kotlin/Java API. Explore its use in Google AI Edge Gallery for advanced AI applications.

- Repository: [google-ai-edge/gallery](https://github.com/google-ai-edge/gallery)
- Tags: deep-dive
- Published: 2026-04-06

---

**LiteRT (Lightweight Runtime) is a minimal-overhead inference engine that enables on-device execution of large language models through a thin Kotlin/Java API, serving as the core backend for chat, multimodal reasoning, and agent skills in the Google AI Edge Gallery app.**

LiteRT powers every LLM-based feature in the Google AI Edge Gallery by abstracting complex model loading, hardware acceleration, and token streaming into a lightweight runtime with a low memory footprint. The runtime is accessed via the `com.google.ai.edge.litertlm.*` package and provides generic interfaces that allow the Gallery app to work with any model converted to the `.litertlm` format.

## Core Architecture of LiteRT

LiteRT is designed specifically for edge deployment, prioritizing binary size efficiency and minimal resource consumption while supporting CPU, GPU, and NPU backends.

### The LiteRT API Structure

At the heart of the runtime are two primary classes: `Engine` and `Conversation`. The `Engine` class manages the native inference runtime and hardware backend initialization, while `Conversation` handles stateful dialogue management, including system instructions, tool definitions, and multimodal context. All interactions flow through the `LlmModelHelper` interface in the Gallery codebase, which wraps LiteRT-specific implementations to keep UI components decoupled from runtime details.

## How LiteRT Is Integrated in Google AI Edge Gallery

The Gallery app follows a five-stage pattern to leverage LiteRT for on-device inference, from hardware selection to resource cleanup.

### Backend Selection and Engine Configuration

The app reads each model's `ACCELERATOR` configuration to instantiate the appropriate `Backend` object. Available options include `Backend.CPU()`, `Backend.GPU()`, and `Backend.NPU(nativeLibraryDir = context.applicationInfo.nativeLibraryDir)`. The `EngineConfig` class bundles these backends with model paths, token limits, and cache directories.

In [`LlmChatModelHelper.kt`](https://github.com/google-ai-edge/gallery/blob/main/LlmChatModelHelper.kt), the configuration is built as follows:

- **Vision and Audio Support**: Conditional `visionBackend` and `audioBackend` parameters enable multimodal capabilities when the model supports them.
- **Cache Management**: Temporary models use `context.getExternalFilesDir(null)?.absolutePath` for caching, while permanent installations use the model's own directory.

### Conversation and Inference Loop

Once configured, the `Engine` initializes native resources via `initialize()`, then creates a `Conversation` object through `createConversation()`. This object accepts `ConversationConfig` containing `SamplerConfig` parameters (temperature, topK, topP), system instructions, and tool providers.

Inference runs asynchronously via `sendMessageAsync()`, which streams partial results through a `MessageCallback` interface. The Gallery's `LlmChatViewModel` observes these callbacks to update the UI in real time.

## Implementation Examples from the Source Code

The following patterns are extracted from [`Android/src/app/src/main/java/com/google/ai/edge/gallery/ui/llmchat/LlmChatModelHelper.kt`](https://github.com/google-ai-edge/gallery/blob/main/Android/src/app/src/main/java/com/google/ai/edge/gallery/ui/llmchat/LlmChatModelHelper.kt) and demonstrate production usage of the LiteRT API.

### Initializing the Engine and Creating Conversations

This snippet shows the complete initialization chain from backend selection to conversation creation:

```kotlin
// 1️⃣ Choose back-ends based on model config
val preferredBackend = when (accelerator) {
    "cpu" -> Backend.CPU()
    "gpu" -> Backend.GPU()
    "npu" -> Backend.NPU(nativeLibraryDir = context.applicationInfo.nativeLibraryDir)
    else   -> Backend.CPU()
}

// 2️⃣ Build the engine configuration
val engineConfig = EngineConfig(
    modelPath = modelPath,
    backend = preferredBackend,
    visionBackend = if (supportImage) visionBackend else null,
    audioBackend  = if (supportAudio) Backend.CPU() else null,
    maxNumTokens = maxTokens,
    cacheDir = if (modelPath.startsWith("/data/local/tmp"))
                  context.getExternalFilesDir(null)?.absolutePath
              else null
)

// 3️⃣ Initialise the engine and create a conversation
val engine = Engine(engineConfig).apply { initialize() }
val conversation = engine.createConversation(
    ConversationConfig(
        samplerConfig = SamplerConfig(topK = topK, topP = topP, temperature = temperature),
        systemInstruction = systemInstruction,
        tools = toolProviders
    )
)

```

### Streaming Multimodal Inference

To handle text, image, and audio inputs simultaneously, the Gallery constructs a `Contents` object and invokes asynchronous streaming:

```kotlin
val contents = mutableListOf<Content>()
images.forEach { bitmap -> contents += Content.ImageBytes(bitmap.toPngByteArray()) }
audioClips.forEach { bytes -> contents += Content.AudioBytes(bytes) }
if (prompt.isNotBlank()) contents += Content.Text(prompt)

// Async streaming – UI receives each token via the callback
conversation.sendMessageAsync(
    Contents.of(contents),
    object : MessageCallback {
        override fun onMessage(message: Message) {
            // Partial result (token or thought) arrives here
            resultListener(message.toString(), false, message.channels["thought"])
        }
        override fun onDone() { resultListener("", true, null) }
        override fun onError(t: Throwable) { errorListener(t.message ?: "Unknown error") }
    },
    extraContext = mapOf("enable_thinking" to "true") // optional
)

```

### Conversation Lifecycle Management

When users clear chat history, the Gallery resets the conversation by creating a fresh `Conversation` instance while preserving the existing `Engine`:

```kotlin
engine.createConversation(
    ConversationConfig(
        samplerConfig = SamplerConfig(topK = topK, topP = topP, temperature = temperature),
        systemInstruction = systemInstruction,
        tools = toolProviders
    )
).also { newConversation ->
    // Replace the old conversation with a fresh one
    (model.instance as LlmModelInstance).conversation = newConversation
}

```

## Key Source Files and Architecture

Understanding LiteRT's role requires examining these specific files within the `google-ai-edge/gallery` repository:

- **[`LlmChatModelHelper.kt`](https://github.com/google-ai-edge/gallery/blob/main/LlmChatModelHelper.kt)** (path: `Android/src/app/src/main/java/com/google/ai/edge/gallery/ui/llmchat/`): Contains the core LiteRT integration, including `Engine` initialization, `Conversation` management, and the `sendMessageAsync()` wrapper used by the chat UI.

- **[`LlmChatViewModel.kt`](https://github.com/google-ai-edge/gallery/blob/main/LlmChatViewModel.kt)**: Acts as the intermediary between the UI layer and LiteRT runtime, forwarding user inputs to `LlmChatModelHelper` and managing the observable state of streaming responses.

- **[`LlmModelHelper.kt`](https://github.com/google-ai-edge/gallery/blob/main/LlmModelHelper.kt)**: Defines the abstract interface that allows the Gallery to interact with any model implementation uniformly, regardless of whether it uses LiteRT or other future runtimes.

- **`model_allowlists/*.json`**: JSON configuration files listing models already converted to the `.litertlm` format, specifying supported backends and multimodal capabilities.

## Summary

- **LiteRT** is a Lightweight Runtime that executes LLMs on-device with minimal overhead, exposed through the `com.google.ai.edge.litertlm` Kotlin/Java API.
- The **Google AI Edge Gallery** uses LiteRT as its exclusive backend for chat, multimodal Q&A, and agent skills via the `Engine` and `Conversation` classes.
- **Backend flexibility** allows runtime selection between CPU, GPU, and NPU via `EngineConfig`, with automatic fallback handling.
- **Multimodal support** is native to the API, accepting `Content.ImageBytes`, `Content.AudioBytes`, and `Content.Text` within the same conversation turn.
- **Resource management** follows a clear lifecycle: configure → initialize → create conversation → stream tokens → reset or close.

## Frequently Asked Questions

### What does LiteRT stand for?

LiteRT stands for **Lightweight Runtime**. It is specifically engineered for edge deployment scenarios where binary size, memory usage, and battery efficiency are critical constraints, distinguishing it from heavier server-side inference frameworks.

### Which programming languages does LiteRT support?

The primary API is implemented in **Kotlin and Java** for Android development, as evidenced by the `com.google.ai.edge.litertlm.*` package structure throughout the Gallery codebase. The underlying runtime is native code accessed through JNI bindings.

### What hardware accelerators does LiteRT support?

LiteRT supports **CPU**, **GPU**, and **NPU** (Neural Processing Unit) backends. The Gallery app detects model capabilities and device hardware to instantiate `Backend.CPU()`, `Backend.GPU()`, or `Backend.NPU()`, passing the `nativeLibraryDir` for NPU-specific drivers when required.

### Where can I find models compatible with LiteRT?

Compatible models are listed in the `model_allowlists` directory of the repository, specifically JSON files identifying models converted to the **`.litertlm`** format. These files specify required accelerators and multimodal support levels, allowing the Gallery to filter available models based on device capabilities.