# Google AI Edge Gallery Multimodal Capabilities: On-Device Image and Audio Processing

> Explore multimodal capabilities in the Google AI Edge Gallery. Process images and audio on-device with Ask Image and Audio Scribe tasks for LLMs.

- Repository: [google-ai-edge/gallery](https://github.com/google-ai-edge/gallery)
- Tags: deep-dive
- Published: 2026-04-06

---

**Google AI Edge Gallery supports multimodal capabilities for processing images and audio as inputs to on-device large language models through the Ask Image and Audio Scribe tasks.**

The Google AI Edge Gallery repository (`google-ai-edge/gallery`) implements on-device multimodal inference using the LiteRT runtime. These multimodal capabilities allow users to attach visual and audio content to chat sessions, enabling the underlying LLMs to perform visual understanding, transcription, and translation without network connectivity.

## Core Multimodal Support

The application exposes multimodal capabilities through two primary task types that integrate with the LLM chat interface. Both capabilities execute entirely on-device, ensuring privacy and offline functionality.

### Ask Image – Visual Understanding

The **Ask Image** capability allows users to attach one or more photos (up to **10 per chat session**) and query the model about visual content. This includes object identification, visual puzzle solving, and detailed scene description.

Implementation resides in `LlmAskImageTask` within **[[`LlmChatTaskModule.kt`](https://github.com/google-ai-edge/gallery/blob/main/LlmChatTaskModule.kt)](https://github.com/google-ai-edge/gallery/blob/main/Android/src/app/src/main/java/com/google/ai/edge/gallery/ui/llmchat/LlmChatTaskModule.kt)**. The task is registered via the `LlmAskImageModule` Dagger-Hilt module, which injects the task into the application's `CustomTask` set. When a model is selected, the runtime initialization sets `supportImage = true` (lines 165-167) to load the multimodal vision pipeline.

### Audio Scribe – Speech-to-Text and Translation

The **Audio Scribe** capability records or accepts uploaded audio clips and returns transcriptions or translations entirely on-device. This feature is implemented through `LlmAskAudioTask` in **[[`LlmChatTaskModule.kt`](https://github.com/google-ai-edge/gallery/blob/main/LlmChatTaskModule.kt)](https://github.com/google-ai-edge/gallery/blob/main/Android/src/app/src/main/java/com/google/ai/edge/gallery/ui/llmchat/LlmChatTaskModule.kt)**.

The `LlmAskAudioModule` registers this task, and the runtime configuration sets `supportAudio = true` (lines 30-32) to enable the audio inference pipeline. The system processes audio data as `ByteArray` objects converted to `Content.AudioBytes` before inference.

## Architecture and Implementation

### Task Registration and Dependency Injection

Custom multimodal tasks integrate into the application through Dagger-Hilt modules:

- **`LlmAskImageModule`**: Registers the image task for dependency injection
- **`LlmAskAudioModule`**: Registers the audio task for dependency injection

Both modules insert their respective tasks into the `CustomTask` set, making them available to the LLM chat interface.

### Runtime Configuration

When a user selects a model, the system calls `model.runtimeHelper.initialize` with flags indicating supported modalities. According to the source code in **[`LlmChatTaskModule.kt`](https://github.com/google-ai-edge/gallery/blob/main/LlmChatTaskModule.kt)**, the initialization accepts boolean parameters `supportImage` and `supportAudio` to configure the LiteRT runtime for the appropriate inference pipelines.

### Content Packaging and Model Inference

The **[`LlmChatModelHelper.kt`](https://github.com/google-ai-edge/gallery/blob/main/LlmChatModelHelper.kt)** file handles conversion of UI-provided media into model-compatible formats:

- **Images**: `Bitmap` objects convert to `Content.ImageBytes` via `image.toPngByteArray()`
- **Audio**: `ByteArray` objects wrap as `Content.AudioBytes`

The helper's `runInference` method (line 269) appends these content objects to the prompt before invoking the model.

### UI Components and Input Handling

The multimodal interface lives in **[`LlmChatScreen.kt`](https://github.com/google-ai-edge/gallery/blob/main/LlmChatScreen.kt)**, with input controls defined in **[`MessageInputText.kt`](https://github.com/google-ai-edge/gallery/blob/main/MessageInputText.kt)**. The image picker enforces a maximum of 10 images per chat session (`MAX_IMAGE_COUNT = 10`). Empty-state strings for the Ask Image UI are defined in **[`strings.xml`](https://github.com/google-ai-edge/gallery/blob/main/strings.xml)** under keys `askimage_emptystate_title` and `askimage_emptystate_content`.

### Model Compatibility Filtering

The **[[`model_allowlist.json`](https://github.com/google-ai-edge/gallery/blob/main/model_allowlist.json)](https://github.com/google-ai-edge/gallery/blob/main/model_allowlist.json)** file enumerates compatible tasks through the `taskTypes` field, including `"llm_ask_image"` and `"llm_ask_audio"`. This filtering ensures only models supporting the required multimodal capabilities appear in the task selection interface.

## Code Examples

### Processing Images Programmatically

To invoke image-based inference programmatically:

```kotlin
// Assume selectedModel is initialized and images is a List<Bitmap>
val helper = LlmChatModelHelper(context, coroutineScope)
helper.runInference(
    model = selectedModel,
    prompt = "Describe the contents of these photos",
    images = images,               // Multimodal image input (max 10)
    onResult = { result -> 
        // Handle model response
    }
)

```

The `runInference` method builds the content list, converts each `Bitmap` to `Content.ImageBytes(image.toPngByteArray())`, and executes the LiteRT model.

### Transcribing Audio Content

For audio transcription tasks:

```kotlin
val audioBytes: ByteArray = // Recorded or loaded audio data
helper.runInference(
    model = selectedModel,
    prompt = "",                       // Optional text prompt
    audio = listOf(audioBytes),        // Multimodal audio input
    onResult = { transcript -> 
        // Handle transcription or translation result
    }
)

```

## Summary

- **Google AI Edge Gallery** supports on-device multimodal processing of **images** and **audio** through specialized LLM tasks.
- **Ask Image** (`LlmAskImageTask`) handles visual understanding with support for up to 10 images per chat, implemented in [`LlmChatTaskModule.kt`](https://github.com/google-ai-edge/gallery/blob/main/LlmChatTaskModule.kt).
- **Audio Scribe** (`LlmAskAudioTask`) provides on-device speech-to-text and translation capabilities.
- The **LiteRT runtime** initializes with `supportImage` and `supportAudio` flags to load appropriate inference pipelines.
- **LlmChatModelHelper.kt** converts `Bitmap` and `ByteArray` objects into model-compatible `Content` formats.
- Model compatibility is enforced through **[`model_allowlist.json`](https://github.com/google-ai-edge/gallery/blob/main/model_allowlist.json)**, which filters by `taskTypes` including `"llm_ask_image"` and `"llm_ask_audio"`.

## Frequently Asked Questions

### What types of media does Google AI Edge Gallery support for multimodal input?

The repository supports **images** (via the Ask Image task) and **audio** (via the Audio Scribe task). Images are processed as `Bitmap` objects up to a maximum of 10 per conversation, while audio is handled as `ByteArray` data for transcription or translation tasks.

### How many images can be attached to a single chat session?

The UI enforces a limit of **10 images per chat session** through the `MAX_IMAGE_COUNT` constant in [`MessageInputText.kt`](https://github.com/google-ai-edge/gallery/blob/main/MessageInputText.kt). This limit is checked during the image selection process before content is packaged and sent to the model.

### Is the audio transcription performed on-device or in the cloud?

All audio processing occurs **entirely on-device**. The `LlmAskAudioTask` uses the LiteRT runtime with `supportAudio = true` to execute inference locally, ensuring no network calls transmit audio data to external servers.

### Which source file handles the conversion of media to model-compatible formats?

**[`LlmChatModelHelper.kt`](https://github.com/google-ai-edge/gallery/blob/main/LlmChatModelHelper.kt)** manages content packaging. It converts `Bitmap` images to `Content.ImageBytes` using `toPngByteArray()` and wraps audio `ByteArray` data as `Content.AudioBytes` before appending these to the inference request.