# How LlmChatModelHelper.kt Interfaces with the LiteRT Runtime for On-Device LLM Inference

> Discover how LlmChatModelHelper.kt bridges your chat UI with the LiteRT LLM runtime. Learn about model configuration, backend selection, and native resource management for efficient on-device inference.

- Repository: [google-ai-edge/gallery](https://github.com/google-ai-edge/gallery)
- Tags: deep-dive
- Published: 2026-04-06

---

**LlmChatModelHelper.kt serves as the concrete bridge between the Gallery app's chat UI and the LiteRT LM runtime, translating high-level chat operations into Engine and Conversation API calls while managing model configuration, backend selection, and native resource lifecycle.**

The `LlmChatModelHelper` class in the [google-ai-edge/gallery](https://github.com/google-ai-edge/gallery) repository provides the primary integration point between Android UI components and the on-device Large Language Model capabilities of the LiteRT (`com.google.ai.edge.litertlm`) library. This Kotlin helper implements the abstract `LlmModelHelper` interface to orchestrate model initialization, multimodal inference, and conversation state management. Understanding how this class interfaces with the LiteRT runtime is essential for developers building production chat applications with edge AI deployment.

## Architecture and Interface Contract

`LlmChatModelHelper` fulfills the `LlmModelHelper` contract defined in [`Android/src/app/src/main/java/com/google/ai/edge/gallery/runtime/LlmModelHelper.kt`](https://github.com/google-ai-edge/gallery/blob/main/Android/src/app/src/main/java/com/google/ai/edge/gallery/runtime/LlmModelHelper.kt). This generic interface standardizes five core operations across different runtime implementations: **initialize**, **resetConversation**, **runInference**, **stopResponse**, and **cleanUp**.

By implementing this interface, `LlmChatModelHelper` enables the UI layer to swap between different LLM runtimes (such as TensorFlow Lite or LiteRT LM) without modifying ViewModel code. The helper encapsulates all LiteRT-specific logic, exposing only generic methods that accept app-level `Model` objects and callback interfaces.

## Engine Initialization and Backend Selection

When `LlmChatModelHelper.initialize()` is invoked, the helper executes a multi-step setup process to prepare the LiteRT LM runtime:

1. **Configuration Parsing**: Reads model-specific parameters using `Model.getIntConfigValue()` and `Model.getStringConfigValue()` to extract max tokens, top‑k, top‑p, temperature, and accelerator preferences.
2. **Backend Selection**: Instantiates the appropriate `Backend` object:
   - CPU: `Backend.CPU()`
   - GPU: `Backend.GPU()`
   - NPU: `Backend.NPU(nativeLibraryDir = …)` requiring a path to native libraries
3. **Engine Construction**: Builds an `EngineConfig` pointing to the model file, configures token limits, and sets cache locations before instantiating `Engine(engineConfig)`.

```kotlin
val engine = Engine(engineConfig)
engine.initialize()
val conversation = engine.createConversation(
    ConversationConfig(
        samplerConfig = SamplerConfig(...),
        systemInstruction = systemInstruction,
        tools = tools
    )
)
model.instance = LlmModelInstance(engine, conversation)

```

The resulting `Engine` and `Conversation` pair is wrapped in a `LlmModelInstance` data class and stored in `model.instance` for subsequent operations. This design keeps the heavy model loaded in memory while allowing conversation state to refresh independently.

## Conversation Lifecycle Management

### Resetting Conversations

The `resetConversation()` method preserves the initialized `Engine` while closing the existing `Conversation` via `instance.conversation.close()`. It then creates a fresh `Conversation` with updated `SamplerConfig` values recomputed from the current model configuration. This approach avoids reloading the multi-gigabyte model file while clearing chat history and system instructions.

### Stopping Active Inference

When users interrupt generation, `stopResponse()` calls `conversation.cancelProcess()` to abort the current token generation without destroying the conversation context. This allows immediate UI feedback while maintaining the existing chat session for subsequent messages.

## Inference Execution and Callback Handling

The `runInference()` method translates UI inputs into LiteRT LM `Content` objects and manages asynchronous response streaming:

- **Text**: `Content.Text(input)`
- **Images**: `Content.ImageBytes(image.toPngByteArray())`
- **Audio**: `Content.AudioBytes(audioClip)`

These contents are packaged into a `Contents` object and sent via `conversation.sendMessageAsync()`:

```kotlin
conversation.sendMessageAsync(
    Contents.of(contents),
    object : MessageCallback {
        override fun onMessage(message: Message) {
            resultListener(message.toString(), false, message.channels["thought"])
        }
        override fun onDone() {
            resultListener("", true, null)
        }
        override fun onError(throwable: Throwable) {
            // Distinguishes cancellations from genuine errors
        }
    },
    extraContext ?: emptyMap()
)

```

The `MessageCallback` interface bridges LiteRT responses back to the UI:
- **`onMessage`**: Streams partial text and optional "thought" channel content
- **`onDone`**: Signals generation completion
- **`onError`**: Forwards error states including cancellation distinctions

## Resource Cleanup and Error Handling

The `cleanUp()` method in [`LlmChatModelHelper.kt`](https://github.com/google-ai-edge/gallery/blob/main/LlmChatModelHelper.kt) ensures complete resource release:
1. Closes the `Conversation` (`instance.conversation.close()`)
2. Closes the `Engine` (`instance.engine.close()`)
3. Removes registered `CleanUpListener` callbacks
4. Clears `model.instance` to null

This sequence releases native GPU/NPU handles, memory caches, and file descriptors. The helper also supports optional `CleanUpListener` registration per model name, allowing UI components to execute post-cleanup actions such as logging or analytics.

## Complete Integration Examples

### Initializing the Chat Helper

```kotlin
val helper = LlmChatModelHelper
helper.initialize(
    context = applicationContext,
    model = model,
    supportImage = true,
    supportAudio = false,
    onDone = { errorMsg ->
        if (errorMsg.isEmpty()) {
            // Engine and Conversation ready
        }
    },
    systemInstruction = null,
    tools = emptyList(),
    enableConversationConstrainedDecoding = false,
    coroutineScope = viewModelScope
)

```

### Running Multimodal Inference

```kotlin
helper.runInference(
    model = model,
    input = "Describe this image in detail.",
    resultListener = { partial, done, thinking ->
        if (!done) {
            // Append partial text to UI
        }
    },
    images = listOf(bitmap),
    audioClips = emptyList(),
    coroutineScope = viewModelScope
)

```

### Cleaning Up Resources

```kotlin
override fun onCleared() {
    super.onCleared()
    LlmChatModelHelper.cleanUp(model) {
        // Optional: Update UI after native resources released
    }
}

```

## Summary

- **LlmChatModelHelper.kt** implements the `LlmModelHelper` interface to provide a runtime-agnostic API for the Gallery app's chat features.
- The helper manages **LiteRT LM Engine** lifecycle, including backend selection (CPU, GPU, NPU) and `EngineConfig` construction based on model-specific JSON configurations.
- **Conversation** objects handle stateful chat sessions and can be reset without reloading the base model, preserving memory and startup time.
- Inference converts Android UI types (Bitmap, ByteArray, String) into LiteRT `Content` objects, using `sendMessageAsync()` with `MessageCallback` for streaming responses.
- Resource cleanup explicitly closes both `Conversation` and `Engine` instances to release native GPU/NPU resources and prevent memory leaks.

## Frequently Asked Questions

### What is the difference between LlmChatModelHelper and LlmModelHelper?

**`LlmModelHelper`** is the abstract interface that defines generic operations (initialize, runInference, cleanUp) required by any LLM runtime implementation in the Gallery app. **`LlmChatModelHelper`** is the concrete Kotlin class that implements this interface specifically for the LiteRT LM runtime, handling LiteRT-specific types like `Engine`, `Conversation`, and `SamplerConfig` while translating them into the generic contract expected by `LlmChatViewModel`.

### How does LlmChatModelHelper handle multimodal inputs like images and audio?

The helper converts Android UI types into LiteRT LM `Content` objects within `runInference()`. Images are converted using `Content.ImageBytes(image.toPngByteArray())`, audio clips use `Content.AudioBytes(audioClip)`, and text uses `Content.Text(input)`. These contents are bundled into a `Contents` object and passed to `conversation.sendMessageAsync()`, enabling the LiteRT runtime to process vision and audio alongside text tokens in a single inference call.

### What LiteRT backends are supported by LlmChatModelHelper?

According to the source code in [`LlmChatModelHelper.kt`](https://github.com/google-ai-edge/gallery/blob/main/LlmChatModelHelper.kt), the helper supports three acceleration backends via the LiteRT LM `Backend` class: **CPU** (`Backend.CPU()`), **GPU** (`Backend.GPU()`), and **NPU** (`Backend.NPU(nativeLibraryDir = …)`). The NPU backend requires a specific native library directory path for delegate initialization, while CPU and GPU selections depend on model configuration flags and device capabilities.

### How does conversation resetting preserve model resources?

When `resetConversation()` is called, the helper invokes `instance.conversation.close()` to destroy the current conversation state, then immediately creates a new `Conversation` using the existing `Engine` via `engine.createConversation()`. This approach preserves the loaded model weights and engine state in memory while clearing only the transient conversation history and sampler configurations, avoiding the multi-second overhead of reloading large model files from disk.