# How to Run LLMs Entirely On-Device with Google AI Edge Gallery: Complete Implementation Guide

> Run LLMs entirely on-device with Google AI Edge Gallery and the MediaPipe LLM Inference API. Discover a complete implementation guide for local, private model execution.

- Repository: [google-ai-edge/gallery](https://github.com/google-ai-edge/gallery)
- Tags: how-to-guide
- Published: 2026-04-06

---

**Google AI Edge Gallery is an Android application that enables developers to download, initialize, and execute large language models completely on-device using the MediaPipe LLM Inference API, ensuring all inference happens locally without sending user data to external servers.**

Google AI Edge Gallery demonstrates how to run LLMs entirely on-device with Google AI Edge Gallery using LiteRT and MediaPipe. The open-source repository provides a complete reference implementation for Android developers who want to build privacy-preserving AI applications where model execution happens entirely on the device CPU, GPU, or NPU.

## Understanding the On-Device Architecture

The Google AI Edge Gallery app implements a layered architecture that keeps all data processing local. The system relies on several key components working together to manage model discovery, downloading, initialization, and inference.

### Model Discovery and Allow-Listing

At startup, the app loads **[`model_allowlist.json`](https://github.com/google-ai-edge/gallery/blob/main/model_allowlist.json)** from the repository root. This JSON file acts as a catalog of available LLMs, specifying supported accelerators (CPU, GPU, NPU) and multimodal capabilities including image and audio support.

### Model Management Layer

The **`ModelManagerViewModel`** class handles the lifecycle of on-device models. According to the source code in [`Android/src/app/src/main/java/com/google/ai/edge/gallery/ui/modelmanager/ModelManagerViewModel.kt`](https://github.com/google-ai-edge/gallery/blob/main/Android/src/app/src/main/java/com/google/ai/edge/gallery/ui/modelmanager/ModelManagerViewModel.kt), this ViewModel coordinates downloading models to the app's external files directory, tracking download state, and managing model initialization.

### Runtime Helper and LLM Engine

The **`Model.runtimeHelper`** extension property defined in [`ModelHelperExt.kt`](https://github.com/google-ai-edge/gallery/blob/main/ModelHelperExt.kt) returns a concrete **`LlmModelHelper`** implementation. The **`LlmChatModelHelper`** class in [`LlmChatModelHelper.kt`](https://github.com/google-ai-edge/gallery/blob/main/LlmChatModelHelper.kt) creates the MediaPipe LiteRT **`Engine`** with the appropriate backend and manages the **`Conversation`** object that streams tokens back to the UI.

## The On-Device Execution Flow

Running an LLM entirely on-device follows a six-step pipeline that ensures data never leaves the device:

1. **Discovery**: `ModelManagerViewModel.loadModelAllowlist()` parses the JSON catalog and populates available tasks.
2. **Selection**: User selection triggers `selectModel()` to store the choice in UI state.
3. **Download**: `downloadRepository.downloadModel()` streams the model file locally if not present.
4. **Initialization**: `initializeModel()` creates the LiteRT `Engine` through `LlmChatModelHelper.initialize()`, configuring vision or audio backends based on model capabilities.
5. **Inference**: `LlmChatViewModel.generateResponse()` invokes `runtimeHelper.runInference()`, which builds a `Contents` list and calls `conversation.sendMessageAsync()`.
6. **Cleanup**: `cleanupModel()` closes the `Conversation` and native `Engine` to free GPU/CPU memory.

## Implementing On-Device LLM Inference

To integrate this functionality into your own Android application, follow these implementation patterns derived from the Google AI Edge Gallery source code.

### Load the Model Catalog

First, initialize the model manager to discover available LLMs:

```kotlin
val modelManager = viewModel<ModelManagerViewModel>()
modelManager.loadModelAllowlist()

// Select first GPU-accelerated model for LLM_CHAT task
val llmModel = modelManager.uiState
    .value.tasks
    .find { it.id == BuiltInTaskId.LLM_CHAT }
    ?.models?.firstOrNull { 
        it.getStringConfigValue(ConfigKeys.ACCELERATOR, "") == Accelerator.GPU.label 
    }

```

### Download and Initialize Models

Before inference, ensure the model is downloaded and initialized:

```kotlin
llmModel?.let { model ->
    // Download if not cached locally
    modelManager.downloadModel(task = model.task, model = model)
    
    // Initialize engine with selected accelerator
    modelManager.initializeModel(
        context = applicationContext,
        task = model.task,
        model = model,
        onDone = { Log.d("Demo", "Model ${model.name} ready") }
    )
}

```

### Execute Text-Only Inference

Run prompts without network connectivity:

```kotlin
val llmChatVm = viewModel<LlmChatViewModel>()

llmChatVm.generateResponse(
    model = llmModel!!,
    input = "Explain quantum computing in simple terms",
    onError = { err -> Log.e("Demo", "LLM error: $err") },
    allowThinking = true  // Enables streaming thought process
)

```

### Process Multimodal Inputs

Handle image and text inputs locally:

```kotlin
val bitmap: Bitmap = // Load from gallery or camera

llmChatVm.generateResponse(
    model = llmModel!!,
    input = "Describe what you see in this image",
    images = listOf(bitmap),
    onError = { err -> Log.e("Demo", err) }
)

```

### Manage Resources

Release memory when switching models or exiting:

```kotlin
modelManager.cleanupModel(
    context = applicationContext, 
    task = llmModel.task, 
    model = llmModel
)

```

## Key Source Files Reference

Understanding these files is essential for customizing the on-device LLM implementation:

- **[`model_allowlist.json`](https://github.com/google-ai-edge/gallery/blob/main/model_allowlist.json)**: JSON catalog defining available models, accelerators, and capabilities
- **[`ModelManagerViewModel.kt`](https://github.com/google-ai-edge/gallery/blob/main/ModelManagerViewModel.kt)**: Handles download, storage, and initialization states
- **[`ModelHelperExt.kt`](https://github.com/google-ai-edge/gallery/blob/main/ModelHelperExt.kt)**: Extension providing `runtimeHelper` access for `Model` objects
- **[`LlmChatModelHelper.kt`](https://github.com/google-ai-edge/gallery/blob/main/LlmChatModelHelper.kt)**: Core LiteRT wrapper managing `Engine` and `Conversation` lifecycle
- **[`LlmChatViewModel.kt`](https://github.com/google-ai-edge/gallery/blob/main/LlmChatViewModel.kt)**: UI layer building request payloads and handling streaming responses
- **[`LlmChatTaskModule.kt`](https://github.com/google-ai-edge/gallery/blob/main/LlmChatTaskModule.kt)**: Dependency injection wiring UI components to ViewModels

## Summary

- Google AI Edge Gallery demonstrates **complete on-device LLM execution** using MediaPipe LiteRT without cloud dependencies.
- The architecture separates concerns across **model discovery** ([`model_allowlist.json`](https://github.com/google-ai-edge/gallery/blob/main/model_allowlist.json)), **management** (`ModelManagerViewModel`), and **inference** (`LlmChatModelHelper`).
- All user data remains on the device; network usage is limited to **optional model downloads** from the allow-list URL.
- The implementation supports **multimodal inputs** (text, image, audio) and hardware acceleration via CPU, GPU, or NPU.
- Proper resource management requires calling `cleanupModel()` to release native `Engine` and `Conversation` objects.

## Frequently Asked Questions

### What hardware accelerators does Google AI Edge Gallery support?

The application supports CPU, GPU, and NPU (Neural Processing Unit) acceleration as defined in [`model_allowlist.json`](https://github.com/google-ai-edge/gallery/blob/main/model_allowlist.json). Each model entry specifies compatible accelerators through configuration keys, and the `LlmChatModelHelper` initializes the LiteRT `Engine` with the appropriate backend during model setup.

### Does Google AI Edge Gallery send my prompts to cloud servers?

No. All inference happens entirely on-device using the MediaPipe LLM Inference API. The only network traffic occurs when downloading model files from the URLs specified in the allow-list. Once downloaded, all text generation, image analysis, and audio processing execute locally within the Android app's sandbox.

### How do I add custom LLM models to run on-device?

You can extend the [`model_allowlist.json`](https://github.com/google-ai-edge/gallery/blob/main/model_allowlist.json) with new model entries specifying the download URL, supported accelerators, and capabilities (image/audio support). Alternatively, the `ModelManagerViewModel` supports importing user-provided model files through its model creation methods, though you must ensure the models are compatible with LiteRT and the MediaPipe LLM Inference API.

### What is the difference between LLM_CHAT and single-turn modes?

The repository provides `LlmChatViewModel` for multi-turn conversational interfaces that maintain context across messages, and `LlmSingleTurnViewModel` for stateless one-shot queries. Both use the same underlying `LlmChatModelHelper` engine but differ in how they manage conversation history and UI state.