Google AI Edge Gallery Multimodal Capabilities: On-Device Image and Audio Processing

Google AI Edge Gallery supports multimodal capabilities for processing images and audio as inputs to on-device large language models through the Ask Image and Audio Scribe tasks.

The Google AI Edge Gallery repository (google-ai-edge/gallery) implements on-device multimodal inference using the LiteRT runtime. These multimodal capabilities allow users to attach visual and audio content to chat sessions, enabling the underlying LLMs to perform visual understanding, transcription, and translation without network connectivity.

Core Multimodal Support

The application exposes multimodal capabilities through two primary task types that integrate with the LLM chat interface. Both capabilities execute entirely on-device, ensuring privacy and offline functionality.

Ask Image – Visual Understanding

The Ask Image capability allows users to attach one or more photos (up to 10 per chat session) and query the model about visual content. This includes object identification, visual puzzle solving, and detailed scene description.

Implementation resides in LlmAskImageTask within [LlmChatTaskModule.kt](https://github.com/google-ai-edge/gallery/blob/main/Android/src/app/src/main/java/com/google/ai/edge/gallery/ui/llmchat/LlmChatTaskModule.kt). The task is registered via the LlmAskImageModule Dagger-Hilt module, which injects the task into the application's CustomTask set. When a model is selected, the runtime initialization sets supportImage = true (lines 165-167) to load the multimodal vision pipeline.

Audio Scribe – Speech-to-Text and Translation

The Audio Scribe capability records or accepts uploaded audio clips and returns transcriptions or translations entirely on-device. This feature is implemented through LlmAskAudioTask in [LlmChatTaskModule.kt](https://github.com/google-ai-edge/gallery/blob/main/Android/src/app/src/main/java/com/google/ai/edge/gallery/ui/llmchat/LlmChatTaskModule.kt).

The LlmAskAudioModule registers this task, and the runtime configuration sets supportAudio = true (lines 30-32) to enable the audio inference pipeline. The system processes audio data as ByteArray objects converted to Content.AudioBytes before inference.

Architecture and Implementation

Task Registration and Dependency Injection

Custom multimodal tasks integrate into the application through Dagger-Hilt modules:

• LlmAskImageModule: Registers the image task for dependency injection
• LlmAskAudioModule: Registers the audio task for dependency injection

Both modules insert their respective tasks into the CustomTask set, making them available to the LLM chat interface.

Runtime Configuration

When a user selects a model, the system calls model.runtimeHelper.initialize with flags indicating supported modalities. According to the source code in LlmChatTaskModule.kt, the initialization accepts boolean parameters supportImage and supportAudio to configure the LiteRT runtime for the appropriate inference pipelines.

Content Packaging and Model Inference

The LlmChatModelHelper.kt file handles conversion of UI-provided media into model-compatible formats:

• Images: Bitmap objects convert to Content.ImageBytes via image.toPngByteArray()
• Audio: ByteArray objects wrap as Content.AudioBytes

The helper's runInference method (line 269) appends these content objects to the prompt before invoking the model.

UI Components and Input Handling

The multimodal interface lives in LlmChatScreen.kt, with input controls defined in MessageInputText.kt. The image picker enforces a maximum of 10 images per chat session (MAX_IMAGE_COUNT = 10). Empty-state strings for the Ask Image UI are defined in strings.xml under keys askimage_emptystate_title and askimage_emptystate_content.

Model Compatibility Filtering

The [model_allowlist.json](https://github.com/google-ai-edge/gallery/blob/main/model_allowlist.json) file enumerates compatible tasks through the taskTypes field, including "llm_ask_image" and "llm_ask_audio". This filtering ensures only models supporting the required multimodal capabilities appear in the task selection interface.

Code Examples

Processing Images Programmatically

To invoke image-based inference programmatically:

// Assume selectedModel is initialized and images is a List<Bitmap>
val helper = LlmChatModelHelper(context, coroutineScope)
helper.runInference(
    model = selectedModel,
    prompt = "Describe the contents of these photos",
    images = images,               // Multimodal image input (max 10)
    onResult = { result -> 
        // Handle model response
    }
)

The runInference method builds the content list, converts each Bitmap to Content.ImageBytes(image.toPngByteArray()), and executes the LiteRT model.

Transcribing Audio Content

For audio transcription tasks:

val audioBytes: ByteArray = // Recorded or loaded audio data
helper.runInference(
    model = selectedModel,
    prompt = "",                       // Optional text prompt
    audio = listOf(audioBytes),        // Multimodal audio input
    onResult = { transcript -> 
        // Handle transcription or translation result
    }
)

Summary

• Google AI Edge Gallery supports on-device multimodal processing of images and audio through specialized LLM tasks.
• Ask Image (LlmAskImageTask) handles visual understanding with support for up to 10 images per chat, implemented in LlmChatTaskModule.kt.
• Audio Scribe (LlmAskAudioTask) provides on-device speech-to-text and translation capabilities.
• The LiteRT runtime initializes with supportImage and supportAudio flags to load appropriate inference pipelines.
• LlmChatModelHelper.kt converts Bitmap and ByteArray objects into model-compatible Content formats.
• Model compatibility is enforced through model_allowlist.json, which filters by taskTypes including "llm_ask_image" and "llm_ask_audio".

Frequently Asked Questions

What types of media does Google AI Edge Gallery support for multimodal input?

The repository supports images (via the Ask Image task) and audio (via the Audio Scribe task). Images are processed as Bitmap objects up to a maximum of 10 per conversation, while audio is handled as ByteArray data for transcription or translation tasks.

How many images can be attached to a single chat session?

The UI enforces a limit of 10 images per chat session through the MAX_IMAGE_COUNT constant in MessageInputText.kt. This limit is checked during the image selection process before content is packaged and sent to the model.

Is the audio transcription performed on-device or in the cloud?

All audio processing occurs entirely on-device. The LlmAskAudioTask uses the LiteRT runtime with supportAudio = true to execute inference locally, ensuring no network calls transmit audio data to external servers.

Which source file handles the conversion of media to model-compatible formats?

LlmChatModelHelper.kt manages content packaging. It converts Bitmap images to Content.ImageBytes using toPngByteArray() and wraps audio ByteArray data as Content.AudioBytes before appending these to the inference request.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}