What Is LiteRT and How It Powers On-Device LLM Inference in Google AI Edge Gallery

LiteRT (Lightweight Runtime) is a minimal-overhead inference engine that enables on-device execution of large language models through a thin Kotlin/Java API, serving as the core backend for chat, multimodal reasoning, and agent skills in the Google AI Edge Gallery app.

LiteRT powers every LLM-based feature in the Google AI Edge Gallery by abstracting complex model loading, hardware acceleration, and token streaming into a lightweight runtime with a low memory footprint. The runtime is accessed via the com.google.ai.edge.litertlm.* package and provides generic interfaces that allow the Gallery app to work with any model converted to the .litertlm format.

Core Architecture of LiteRT

LiteRT is designed specifically for edge deployment, prioritizing binary size efficiency and minimal resource consumption while supporting CPU, GPU, and NPU backends.

The LiteRT API Structure

At the heart of the runtime are two primary classes: Engine and Conversation. The Engine class manages the native inference runtime and hardware backend initialization, while Conversation handles stateful dialogue management, including system instructions, tool definitions, and multimodal context. All interactions flow through the LlmModelHelper interface in the Gallery codebase, which wraps LiteRT-specific implementations to keep UI components decoupled from runtime details.

How LiteRT Is Integrated in Google AI Edge Gallery

The Gallery app follows a five-stage pattern to leverage LiteRT for on-device inference, from hardware selection to resource cleanup.

Backend Selection and Engine Configuration

The app reads each model's ACCELERATOR configuration to instantiate the appropriate Backend object. Available options include Backend.CPU(), Backend.GPU(), and Backend.NPU(nativeLibraryDir = context.applicationInfo.nativeLibraryDir). The EngineConfig class bundles these backends with model paths, token limits, and cache directories.

In LlmChatModelHelper.kt, the configuration is built as follows:

• Vision and Audio Support: Conditional visionBackend and audioBackend parameters enable multimodal capabilities when the model supports them.
• Cache Management: Temporary models use context.getExternalFilesDir(null)?.absolutePath for caching, while permanent installations use the model's own directory.

Conversation and Inference Loop

Once configured, the Engine initializes native resources via initialize(), then creates a Conversation object through createConversation(). This object accepts ConversationConfig containing SamplerConfig parameters (temperature, topK, topP), system instructions, and tool providers.

Inference runs asynchronously via sendMessageAsync(), which streams partial results through a MessageCallback interface. The Gallery's LlmChatViewModel observes these callbacks to update the UI in real time.

Implementation Examples from the Source Code

The following patterns are extracted from Android/src/app/src/main/java/com/google/ai/edge/gallery/ui/llmchat/LlmChatModelHelper.kt and demonstrate production usage of the LiteRT API.

Initializing the Engine and Creating Conversations

This snippet shows the complete initialization chain from backend selection to conversation creation:

// 1️⃣ Choose back-ends based on model config
val preferredBackend = when (accelerator) {
    "cpu" -> Backend.CPU()
    "gpu" -> Backend.GPU()
    "npu" -> Backend.NPU(nativeLibraryDir = context.applicationInfo.nativeLibraryDir)
    else   -> Backend.CPU()
}

// 2️⃣ Build the engine configuration
val engineConfig = EngineConfig(
    modelPath = modelPath,
    backend = preferredBackend,
    visionBackend = if (supportImage) visionBackend else null,
    audioBackend  = if (supportAudio) Backend.CPU() else null,
    maxNumTokens = maxTokens,
    cacheDir = if (modelPath.startsWith("/data/local/tmp"))
                  context.getExternalFilesDir(null)?.absolutePath
              else null
)

// 3️⃣ Initialise the engine and create a conversation
val engine = Engine(engineConfig).apply { initialize() }
val conversation = engine.createConversation(
    ConversationConfig(
        samplerConfig = SamplerConfig(topK = topK, topP = topP, temperature = temperature),
        systemInstruction = systemInstruction,
        tools = toolProviders
    )
)

Streaming Multimodal Inference

To handle text, image, and audio inputs simultaneously, the Gallery constructs a Contents object and invokes asynchronous streaming:

val contents = mutableListOf<Content>()
images.forEach { bitmap -> contents += Content.ImageBytes(bitmap.toPngByteArray()) }
audioClips.forEach { bytes -> contents += Content.AudioBytes(bytes) }
if (prompt.isNotBlank()) contents += Content.Text(prompt)

// Async streaming – UI receives each token via the callback
conversation.sendMessageAsync(
    Contents.of(contents),
    object : MessageCallback {
        override fun onMessage(message: Message) {
            // Partial result (token or thought) arrives here
            resultListener(message.toString(), false, message.channels["thought"])
        }
        override fun onDone() { resultListener("", true, null) }
        override fun onError(t: Throwable) { errorListener(t.message ?: "Unknown error") }
    },
    extraContext = mapOf("enable_thinking" to "true") // optional
)

Conversation Lifecycle Management

When users clear chat history, the Gallery resets the conversation by creating a fresh Conversation instance while preserving the existing Engine:

engine.createConversation(
    ConversationConfig(
        samplerConfig = SamplerConfig(topK = topK, topP = topP, temperature = temperature),
        systemInstruction = systemInstruction,
        tools = toolProviders
    )
).also { newConversation ->
    // Replace the old conversation with a fresh one
    (model.instance as LlmModelInstance).conversation = newConversation
}

Key Source Files and Architecture

Understanding LiteRT's role requires examining these specific files within the google-ai-edge/gallery repository:

• LlmChatModelHelper.kt (path: Android/src/app/src/main/java/com/google/ai/edge/gallery/ui/llmchat/): Contains the core LiteRT integration, including Engine initialization, Conversation management, and the sendMessageAsync() wrapper used by the chat UI.

• LlmChatViewModel.kt: Acts as the intermediary between the UI layer and LiteRT runtime, forwarding user inputs to LlmChatModelHelper and managing the observable state of streaming responses.

• LlmModelHelper.kt: Defines the abstract interface that allows the Gallery to interact with any model implementation uniformly, regardless of whether it uses LiteRT or other future runtimes.

• model_allowlists/*.json: JSON configuration files listing models already converted to the .litertlm format, specifying supported backends and multimodal capabilities.

Summary

• LiteRT is a Lightweight Runtime that executes LLMs on-device with minimal overhead, exposed through the com.google.ai.edge.litertlm Kotlin/Java API.
• The Google AI Edge Gallery uses LiteRT as its exclusive backend for chat, multimodal Q&A, and agent skills via the Engine and Conversation classes.
• Backend flexibility allows runtime selection between CPU, GPU, and NPU via EngineConfig, with automatic fallback handling.
• Multimodal support is native to the API, accepting Content.ImageBytes, Content.AudioBytes, and Content.Text within the same conversation turn.
• Resource management follows a clear lifecycle: configure → initialize → create conversation → stream tokens → reset or close.

Frequently Asked Questions

What does LiteRT stand for?

LiteRT stands for Lightweight Runtime. It is specifically engineered for edge deployment scenarios where binary size, memory usage, and battery efficiency are critical constraints, distinguishing it from heavier server-side inference frameworks.

Which programming languages does LiteRT support?

The primary API is implemented in Kotlin and Java for Android development, as evidenced by the com.google.ai.edge.litertlm.* package structure throughout the Gallery codebase. The underlying runtime is native code accessed through JNI bindings.

What hardware accelerators does LiteRT support?

LiteRT supports CPU, GPU, and NPU (Neural Processing Unit) backends. The Gallery app detects model capabilities and device hardware to instantiate Backend.CPU(), Backend.GPU(), or Backend.NPU(), passing the nativeLibraryDir for NPU-specific drivers when required.

Where can I find models compatible with LiteRT?

Compatible models are listed in the model_allowlists directory of the repository, specifically JSON files identifying models converted to the .litertlm format. These files specify required accelerators and multimodal support levels, allowing the Gallery to filter available models based on device capabilities.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}