How to Run LLMs Entirely On-Device with Google AI Edge Gallery: Complete Implementation Guide

Google AI Edge Gallery is an Android application that enables developers to download, initialize, and execute large language models completely on-device using the MediaPipe LLM Inference API, ensuring all inference happens locally without sending user data to external servers.

Google AI Edge Gallery demonstrates how to run LLMs entirely on-device with Google AI Edge Gallery using LiteRT and MediaPipe. The open-source repository provides a complete reference implementation for Android developers who want to build privacy-preserving AI applications where model execution happens entirely on the device CPU, GPU, or NPU.

Understanding the On-Device Architecture

The Google AI Edge Gallery app implements a layered architecture that keeps all data processing local. The system relies on several key components working together to manage model discovery, downloading, initialization, and inference.

Model Discovery and Allow-Listing

At startup, the app loads model_allowlist.json from the repository root. This JSON file acts as a catalog of available LLMs, specifying supported accelerators (CPU, GPU, NPU) and multimodal capabilities including image and audio support.

Model Management Layer

The ModelManagerViewModel class handles the lifecycle of on-device models. According to the source code in Android/src/app/src/main/java/com/google/ai/edge/gallery/ui/modelmanager/ModelManagerViewModel.kt, this ViewModel coordinates downloading models to the app's external files directory, tracking download state, and managing model initialization.

Runtime Helper and LLM Engine

The Model.runtimeHelper extension property defined in ModelHelperExt.kt returns a concrete LlmModelHelper implementation. The LlmChatModelHelper class in LlmChatModelHelper.kt creates the MediaPipe LiteRT Engine with the appropriate backend and manages the Conversation object that streams tokens back to the UI.

The On-Device Execution Flow

Running an LLM entirely on-device follows a six-step pipeline that ensures data never leaves the device:

1. Discovery: ModelManagerViewModel.loadModelAllowlist() parses the JSON catalog and populates available tasks.
2. Selection: User selection triggers selectModel() to store the choice in UI state.
3. Download: downloadRepository.downloadModel() streams the model file locally if not present.
4. Initialization: initializeModel() creates the LiteRT Engine through LlmChatModelHelper.initialize(), configuring vision or audio backends based on model capabilities.
5. Inference: LlmChatViewModel.generateResponse() invokes runtimeHelper.runInference(), which builds a Contents list and calls conversation.sendMessageAsync().
6. Cleanup: cleanupModel() closes the Conversation and native Engine to free GPU/CPU memory.

Implementing On-Device LLM Inference

To integrate this functionality into your own Android application, follow these implementation patterns derived from the Google AI Edge Gallery source code.

Load the Model Catalog

First, initialize the model manager to discover available LLMs:

val modelManager = viewModel<ModelManagerViewModel>()
modelManager.loadModelAllowlist()

// Select first GPU-accelerated model for LLM_CHAT task
val llmModel = modelManager.uiState
    .value.tasks
    .find { it.id == BuiltInTaskId.LLM_CHAT }
    ?.models?.firstOrNull { 
        it.getStringConfigValue(ConfigKeys.ACCELERATOR, "") == Accelerator.GPU.label 
    }

Download and Initialize Models

Before inference, ensure the model is downloaded and initialized:

llmModel?.let { model ->
    // Download if not cached locally
    modelManager.downloadModel(task = model.task, model = model)
    
    // Initialize engine with selected accelerator
    modelManager.initializeModel(
        context = applicationContext,
        task = model.task,
        model = model,
        onDone = { Log.d("Demo", "Model ${model.name} ready") }
    )
}

Execute Text-Only Inference

Run prompts without network connectivity:

val llmChatVm = viewModel<LlmChatViewModel>()

llmChatVm.generateResponse(
    model = llmModel!!,
    input = "Explain quantum computing in simple terms",
    onError = { err -> Log.e("Demo", "LLM error: $err") },
    allowThinking = true  // Enables streaming thought process
)

Process Multimodal Inputs

Handle image and text inputs locally:

val bitmap: Bitmap = // Load from gallery or camera

llmChatVm.generateResponse(
    model = llmModel!!,
    input = "Describe what you see in this image",
    images = listOf(bitmap),
    onError = { err -> Log.e("Demo", err) }
)

Manage Resources

Release memory when switching models or exiting:

modelManager.cleanupModel(
    context = applicationContext, 
    task = llmModel.task, 
    model = llmModel
)

Key Source Files Reference

Understanding these files is essential for customizing the on-device LLM implementation:

• model_allowlist.json: JSON catalog defining available models, accelerators, and capabilities
• ModelManagerViewModel.kt: Handles download, storage, and initialization states
• ModelHelperExt.kt: Extension providing runtimeHelper access for Model objects
• LlmChatModelHelper.kt: Core LiteRT wrapper managing Engine and Conversation lifecycle
• LlmChatViewModel.kt: UI layer building request payloads and handling streaming responses
• LlmChatTaskModule.kt: Dependency injection wiring UI components to ViewModels

Summary

• Google AI Edge Gallery demonstrates complete on-device LLM execution using MediaPipe LiteRT without cloud dependencies.
• The architecture separates concerns across model discovery (model_allowlist.json), management (ModelManagerViewModel), and inference (LlmChatModelHelper).
• All user data remains on the device; network usage is limited to optional model downloads from the allow-list URL.
• The implementation supports multimodal inputs (text, image, audio) and hardware acceleration via CPU, GPU, or NPU.
• Proper resource management requires calling cleanupModel() to release native Engine and Conversation objects.

Frequently Asked Questions

What hardware accelerators does Google AI Edge Gallery support?

The application supports CPU, GPU, and NPU (Neural Processing Unit) acceleration as defined in model_allowlist.json. Each model entry specifies compatible accelerators through configuration keys, and the LlmChatModelHelper initializes the LiteRT Engine with the appropriate backend during model setup.

Does Google AI Edge Gallery send my prompts to cloud servers?

No. All inference happens entirely on-device using the MediaPipe LLM Inference API. The only network traffic occurs when downloading model files from the URLs specified in the allow-list. Once downloaded, all text generation, image analysis, and audio processing execute locally within the Android app's sandbox.

How do I add custom LLM models to run on-device?

You can extend the model_allowlist.json with new model entries specifying the download URL, supported accelerators, and capabilities (image/audio support). Alternatively, the ModelManagerViewModel supports importing user-provided model files through its model creation methods, though you must ensure the models are compatible with LiteRT and the MediaPipe LLM Inference API.

What is the difference between LLM_CHAT and single-turn modes?

The repository provides LlmChatViewModel for multi-turn conversational interfaces that maintain context across messages, and LlmSingleTurnViewModel for stateless one-shot queries. Both use the same underlying LlmChatModelHelper engine but differ in how they manage conversation history and UI state.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}