How LlmChatModelHelper.kt Interfaces with the LiteRT Runtime for On-Device LLM Inference

LlmChatModelHelper.kt serves as the concrete bridge between the Gallery app's chat UI and the LiteRT LM runtime, translating high-level chat operations into Engine and Conversation API calls while managing model configuration, backend selection, and native resource lifecycle.

The LlmChatModelHelper class in the google-ai-edge/gallery repository provides the primary integration point between Android UI components and the on-device Large Language Model capabilities of the LiteRT (com.google.ai.edge.litertlm) library. This Kotlin helper implements the abstract LlmModelHelper interface to orchestrate model initialization, multimodal inference, and conversation state management. Understanding how this class interfaces with the LiteRT runtime is essential for developers building production chat applications with edge AI deployment.

Architecture and Interface Contract

LlmChatModelHelper fulfills the LlmModelHelper contract defined in Android/src/app/src/main/java/com/google/ai/edge/gallery/runtime/LlmModelHelper.kt. This generic interface standardizes five core operations across different runtime implementations: initialize, resetConversation, runInference, stopResponse, and cleanUp.

By implementing this interface, LlmChatModelHelper enables the UI layer to swap between different LLM runtimes (such as TensorFlow Lite or LiteRT LM) without modifying ViewModel code. The helper encapsulates all LiteRT-specific logic, exposing only generic methods that accept app-level Model objects and callback interfaces.

Engine Initialization and Backend Selection

When LlmChatModelHelper.initialize() is invoked, the helper executes a multi-step setup process to prepare the LiteRT LM runtime:

1. Configuration Parsing: Reads model-specific parameters using Model.getIntConfigValue() and Model.getStringConfigValue() to extract max tokens, top‑k, top‑p, temperature, and accelerator preferences.
2. Backend Selection: Instantiates the appropriate Backend object:
   • CPU: Backend.CPU()
   • GPU: Backend.GPU()
   • NPU: Backend.NPU(nativeLibraryDir = …) requiring a path to native libraries
3. Engine Construction: Builds an EngineConfig pointing to the model file, configures token limits, and sets cache locations before instantiating Engine(engineConfig).

val engine = Engine(engineConfig)
engine.initialize()
val conversation = engine.createConversation(
    ConversationConfig(
        samplerConfig = SamplerConfig(...),
        systemInstruction = systemInstruction,
        tools = tools
    )
)
model.instance = LlmModelInstance(engine, conversation)

The resulting Engine and Conversation pair is wrapped in a LlmModelInstance data class and stored in model.instance for subsequent operations. This design keeps the heavy model loaded in memory while allowing conversation state to refresh independently.

Conversation Lifecycle Management

Resetting Conversations

The resetConversation() method preserves the initialized Engine while closing the existing Conversation via instance.conversation.close(). It then creates a fresh Conversation with updated SamplerConfig values recomputed from the current model configuration. This approach avoids reloading the multi-gigabyte model file while clearing chat history and system instructions.

Stopping Active Inference

When users interrupt generation, stopResponse() calls conversation.cancelProcess() to abort the current token generation without destroying the conversation context. This allows immediate UI feedback while maintaining the existing chat session for subsequent messages.

Inference Execution and Callback Handling

The runInference() method translates UI inputs into LiteRT LM Content objects and manages asynchronous response streaming:

• Text: Content.Text(input)
• Images: Content.ImageBytes(image.toPngByteArray())
• Audio: Content.AudioBytes(audioClip)

These contents are packaged into a Contents object and sent via conversation.sendMessageAsync():

conversation.sendMessageAsync(
    Contents.of(contents),
    object : MessageCallback {
        override fun onMessage(message: Message) {
            resultListener(message.toString(), false, message.channels["thought"])
        }
        override fun onDone() {
            resultListener("", true, null)
        }
        override fun onError(throwable: Throwable) {
            // Distinguishes cancellations from genuine errors
        }
    },
    extraContext ?: emptyMap()
)

The MessageCallback interface bridges LiteRT responses back to the UI:

• onMessage: Streams partial text and optional "thought" channel content
• onDone: Signals generation completion
• onError: Forwards error states including cancellation distinctions

Resource Cleanup and Error Handling

The cleanUp() method in LlmChatModelHelper.kt ensures complete resource release:

1. Closes the Conversation (instance.conversation.close())
2. Closes the Engine (instance.engine.close())
3. Removes registered CleanUpListener callbacks
4. Clears model.instance to null

This sequence releases native GPU/NPU handles, memory caches, and file descriptors. The helper also supports optional CleanUpListener registration per model name, allowing UI components to execute post-cleanup actions such as logging or analytics.

Complete Integration Examples

Initializing the Chat Helper

val helper = LlmChatModelHelper
helper.initialize(
    context = applicationContext,
    model = model,
    supportImage = true,
    supportAudio = false,
    onDone = { errorMsg ->
        if (errorMsg.isEmpty()) {
            // Engine and Conversation ready
        }
    },
    systemInstruction = null,
    tools = emptyList(),
    enableConversationConstrainedDecoding = false,
    coroutineScope = viewModelScope
)

Running Multimodal Inference

helper.runInference(
    model = model,
    input = "Describe this image in detail.",
    resultListener = { partial, done, thinking ->
        if (!done) {
            // Append partial text to UI
        }
    },
    images = listOf(bitmap),
    audioClips = emptyList(),
    coroutineScope = viewModelScope
)

Cleaning Up Resources

override fun onCleared() {
    super.onCleared()
    LlmChatModelHelper.cleanUp(model) {
        // Optional: Update UI after native resources released
    }
}

Summary

• LlmChatModelHelper.kt implements the LlmModelHelper interface to provide a runtime-agnostic API for the Gallery app's chat features.
• The helper manages LiteRT LM Engine lifecycle, including backend selection (CPU, GPU, NPU) and EngineConfig construction based on model-specific JSON configurations.
• Conversation objects handle stateful chat sessions and can be reset without reloading the base model, preserving memory and startup time.
• Inference converts Android UI types (Bitmap, ByteArray, String) into LiteRT Content objects, using sendMessageAsync() with MessageCallback for streaming responses.
• Resource cleanup explicitly closes both Conversation and Engine instances to release native GPU/NPU resources and prevent memory leaks.

Frequently Asked Questions

What is the difference between LlmChatModelHelper and LlmModelHelper?

LlmModelHelper is the abstract interface that defines generic operations (initialize, runInference, cleanUp) required by any LLM runtime implementation in the Gallery app. LlmChatModelHelper is the concrete Kotlin class that implements this interface specifically for the LiteRT LM runtime, handling LiteRT-specific types like Engine, Conversation, and SamplerConfig while translating them into the generic contract expected by LlmChatViewModel.

How does LlmChatModelHelper handle multimodal inputs like images and audio?

The helper converts Android UI types into LiteRT LM Content objects within runInference(). Images are converted using Content.ImageBytes(image.toPngByteArray()), audio clips use Content.AudioBytes(audioClip), and text uses Content.Text(input). These contents are bundled into a Contents object and passed to conversation.sendMessageAsync(), enabling the LiteRT runtime to process vision and audio alongside text tokens in a single inference call.

What LiteRT backends are supported by LlmChatModelHelper?

According to the source code in LlmChatModelHelper.kt, the helper supports three acceleration backends via the LiteRT LM Backend class: CPU (Backend.CPU()), GPU (Backend.GPU()), and NPU (Backend.NPU(nativeLibraryDir = …)). The NPU backend requires a specific native library directory path for delegate initialization, while CPU and GPU selections depend on model configuration flags and device capabilities.

How does conversation resetting preserve model resources?

When resetConversation() is called, the helper invokes instance.conversation.close() to destroy the current conversation state, then immediately creates a new Conversation using the existing Engine via engine.createConversation(). This approach preserves the loaded model weights and engine state in memory while clearing only the transient conversation history and sampler configurations, avoiding the multi-second overhead of reloading large model files from disk.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}