How to Use the 'Ask Image' Feature for Visual Analysis with the Device Camera in Google AI Edge Gallery

The 'Ask Image' feature in the Google AI Edge Gallery app lets you capture or select photos with your device camera and query on‑device Large Language Models (LLMs) to analyze visual content, answer questions, or solve puzzles without sending data to the cloud.

The google-ai-edge/gallery repository provides a complete Android implementation for running multimodal LLMs locally. By enabling the 'Ask Image' feature for visual analysis with the device camera, developers can build apps where users attach images to chat messages and receive AI-generated descriptions or answers entirely on-device.

How the Ask Image Task is Registered

The feature is implemented as a built-in CustomTask named LlmAskImageTask, registered with the Dagger dependency injection graph via LlmChatTaskModule.

Inside Android/src/app/src/main/java/com/google/ai/edge/gallery/ui/llmchat/LlmChatTaskModule.kt (lines 40‑55), the task metadata defines the user-facing label, icon, and descriptions:

// LlmChatTaskModule.kt – Ask Image task definition
class LlmAskImageTask @Inject constructor() : CustomTask {
  override val task: Task = Task(
    id = BuiltInTaskId.LLM_ASK_IMAGE,
    label = "Ask Image",
    category = Category.LLM,
    icon = Icons.Outlined.Mms,
    models = mutableListOf(),
    description = "Ask questions about images with on‑device large language models",
    shortDescription = "Ask questions about images",
    docUrl = "https://github.com/google-ai-edge/LiteRT-LM/blob/main/kotlin/README.md",
    sourceCodeUrl = "https://github.com/google-ai-edge/gallery/blob/main/Android/src/app/src/main/java/com/google/ai/edge/gallery/ui/llmchat/LlmChatModelHelper.kt",
    textInputPlaceHolderRes = R.string.text_input_placeholder_llm_chat,
  )
  …
}

The task is provided to the Dagger graph using @Provides @IntoSet, making it available throughout the application for navigation and execution.

Enabling Image Support at Runtime

For a model to process visual inputs, the runtime must be initialized with the supportImage flag set to true. In LlmChatTaskModule.kt at line 62, the initialization call looks like this:

// LlmChatTaskModule.kt – model initialization for Ask Image
model.runtimeHelper.initialize(
  context = context,
  model = model,
  supportImage = true,   // <‑‑ crucial for image handling
  supportAudio = false,
  onDone = onDone,
  coroutineScope = coroutineScope,
)

Setting supportImage = true signals the underlying LiteRT‑LM engine to allocate the necessary tensors and preprocessing pipelines for image inputs. Without this flag, the model will reject bitmap inputs even if the UI allows image selection.

Composing the Ask Image User Interface

The UI entry point is LlmAskImageScreen, a composable defined in Android/src/app/src/main/java/com/google/ai/edge/gallery/ui/llmchat/LlmChatScreen.kt (lines 96‑112). This screen wraps the generic ChatViewWrapper but explicitly enables the image picker:

// LlmChatScreen.kt – Ask Image UI entry point
@Composable
fun LlmAskImageScreen(
  modelManagerViewModel: ModelManagerViewModel,
  navigateUp: () -> Unit,
  modifier: Modifier = Modifier,
  viewModel: LlmAskImageViewModel = hiltViewModel(),
) {
  ChatViewWrapper(
    viewModel = viewModel,
    modelManagerViewModel = modelManagerViewModel,
    taskId = BuiltInTaskId.LLM_ASK_IMAGE,
    navigateUp = navigateUp,
    modifier = modifier,
    showImagePicker = true,   // <‑‑ enables the “+” button
    showAudioPicker = false,
    …
  )
}

When showImagePicker = true, the UI displays a + button that launches the camera or gallery picker. The empty‑state guidance shown to users is defined in Android/src/app/src/main/res/values/strings.xml:

<!-- strings.xml – Ask Image empty‑state strings -->
<string name="askimage_emptystate_title" translatable="false">Ask Image</string>
<string name="askimage_emptystate_content" translatable="false">
  To get started, tap the + below to add an image (max 10 images per chat) and type a prompt to ask a question about it!
</string>

Processing Images for Visual Analysis

When the user presses Send, ChatViewWrapper extracts all ChatMessageImage objects from the message list and converts them to Bitmap instances. This logic appears in LlmChatScreen.kt (lines 102‑108):

// LlmChatScreen.kt – extracting images from the message list
val images: MutableList<Bitmap> = mutableListOf()
for (message in messages) {
  if (message is ChatMessageImage) {
    images.addAll(message.bitmaps)   // all selected photos become model inputs
  }
}
viewModel.generateResponse(
  model = model,
  input = text,
  images = images,
  …
)

The generateResponse method passes both the text prompt and the bitmap list to the LLM. The model processes the visual data alongside the query to generate contextual answers.

Filtering Models for Image Compatibility

Not all on‑device models support visual inputs. In Android/src/app/src/main/java/com/google/ai/edge/gallery/ui/modelmanager/ModelManagerViewModel.kt (lines 584‑586), the app filters the model list to show only those compatible with the Ask Image task:

// ModelManagerViewModel – filtering models for Ask Image
(task.id == BuiltInTaskId.LLM_ASK_IMAGE && model.llmSupportImage) ||
(task.id != BuiltInTaskId.LLM_ASK_IMAGE && …)

A model must explicitly declare llmSupportImage = true in its definition to appear as an option for visual analysis. For example:

// Example of a model definition that enables Ask Image
val myVisionModel = Model(
    name = "gemma‑4‑vision",
    url = "https://example.com/gemma-4-vision.tflite",
    // Flags the runtime that this model can take images
    llmSupportImage = true,
    // other fields omitted for brevity …
)

Complete Implementation Example

To integrate the Ask Image flow into your own Compose screen, you can invoke the task programmatically:

@Composable
fun OpenAskImageTaskButton(
    modelManagerViewModel: ModelManagerViewModel,
    navigateUp: () -> Unit
) {
    // The BuiltInTaskId for Ask Image
    val askImageTaskId = BuiltInTaskId.LLM_ASK_IMAGE

    // Find the task (the view model holds the list of tasks)
    val task = modelManagerViewModel.getTaskById(askImageTaskId) ?: return

    // Button that navigates to the Ask Image screen
    Button(onClick = {
        val taskInstance = task.customTask // injected via Dagger
        taskInstance?.MainScreen(
            data = CustomTaskDataForBuiltinTask(
                modelManagerViewModel = modelManagerViewModel,
                onNavUp = navigateUp
            )
        )
    }) {
        Text(text = stringResource(id = R.string.askimage_emptystate_title))
    }
}

To add images programmatically from a camera intent:

// Inside a ViewModel handling the chat UI
fun addImageFromBitmap(bitmap: Bitmap) {
    // Convert bitmap into a ChatMessageImage and push it to the UI message list.
    val imageMessage = ChatMessageImage(bitmaps = listOf(bitmap))
    _uiState.update { it.copy(messages = it.messages + imageMessage) }
}

Summary

• LlmAskImageTask registers the feature as a built-in custom task in LlmChatTaskModule.kt, providing metadata and Dagger injection.
• Runtime initialization requires supportImage = true to enable the LiteRT‑LM engine to process bitmap inputs.
• LlmAskImageScreen wraps ChatViewWrapper with showImagePicker = true, exposing the camera and gallery selection UI.
• Image extraction happens in ChatViewWrapper via ChatMessageImage objects, passing bitmaps to viewModel.generateResponse.
• Model compatibility is enforced by checking model.llmSupportImage in ModelManagerViewModel, ensuring only vision-capable LLMs are presented.

Frequently Asked Questions

How many images can I attach to a single Ask Image query?

According to the empty‑state string in strings.xml, the Google AI Edge Gallery supports up to 10 images per chat. When the user taps the + button, they can add multiple photos from the gallery or take several pictures with the device camera before sending the query.

Does the Ask Image feature work offline?

Yes. The Ask Image feature performs on‑device inference using LiteRT‑LM (Lite Runtime for Language Models). As long as the model is downloaded to the device and initialized with supportImage = true, all visual analysis and text generation happen locally without requiring an internet connection.

What is the difference between taking a picture and picking from the album?

The UI strings defined in strings.xml indicate both options are available: "take a picture" launches the device camera for immediate capture, while "pick from album" opens the photo gallery for selecting existing images. Both methods produce Bitmap objects that are wrapped in ChatMessageImage and processed identically by the generateResponse method.

Which models support visual analysis in the Gallery app?

Only models that explicitly declare llmSupportImage = true in their definition appear in the Ask Image task. The ModelManagerViewModel filters the available models at line 584 to ensure incompatible architectures are hidden. According to the source code, models must also be initialized with supportImage = true in LlmChatTaskModule.kt to enable the image tensor pathways in the LiteRT runtime.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}