How Google AI Edge Gallery Maintains Privacy and Security During On-Device Inference

Google AI Edge Gallery ensures complete user privacy by executing Large Language Models entirely on the device through a sandboxed LiteRT-LM runtime, storing model files in private app directories, and processing all tensor calculations locally without any network connectivity during inference.

The google-ai-edge/gallery repository implements a fully offline architecture for generative AI, guaranteeing that prompts, images, and audio data never leave the local hardware. By leveraging on-device inference, the application eliminates cloud dependencies and prevents external data transmission entirely.

Three-Layer Privacy Architecture for On-Device Inference

The privacy model consists of tightly coupled layers that handle storage, execution, and fallback runtimes.

Secure Model Storage and Isolation

Model files are downloaded once from sources like Hugging Face and stored in the app’s private externalFilesDir. The ModelManagerViewModel class handles this secure download using HttpURLConnection, after which the file resides exclusively in sandboxed storage that other applications cannot access. Once downloaded, the model path is retrieved via model.getPath(context) and passed to the runtime, ensuring the data never leaves the device boundary again.

LiteRT-LM Runtime Execution

The core on-device inference engine is implemented in LlmChatModelHelper.kt. The initialize() method constructs an EngineConfig targeting the LiteRT backend (CPU, GPU, or NPU) and creates a native Engine instance. All tokenization, tensor calculations, and decoding occur inside the LiteRT native library, which contains no network code. As stated in the README, the product is explicitly marketed as "fully offline, private, and lightning‑fast", reflecting that inference happens strictly in-process.

MediaPipe LLM Inference API Fallback

For models requiring alternative execution paths, the app can utilize the MediaPipe LLM Inference API. This pathway creates Engine and Conversation objects that also run completely on-device without exposing network endpoints. Both runtimes provide identical "offline-first" guarantees, ensuring consistent on-device inference regardless of the backend selected.

Architectural Flow of Private Inference

The data pathway demonstrates zero network exposure after initial installation:

1. Secure Download: ModelManagerViewModel fetches the model via HttpURLConnection and writes it to the app-private externalFilesDir.
2. Engine Initialization: LlmChatModelHelper.initialize() loads the model using Engine(engineConfig).initialize() with the LiteRT backend.
3. Local Conversation State: The engine.createConversation() method instantiates a Conversation object that maintains chat history entirely in device memory.
4. Offline Inference: The runInference() method (lines 44‑51 and 79‑91 in LlmChatModelHelper.kt) builds a Contents object and invokes conversation.sendMessageAsync(), which streams results locally without transmitting data to remote servers.

Because the only network interaction is the one-time model download, the app functions completely offline after installation, providing end-to-end privacy for user inputs.

Code Example: Implementing Private On-Device Chat

The following Kotlin implementation demonstrates how to initialize and run inference locally using the LiteRT-LM helper:

// 1. Retrieve model from private repository (already downloaded)
val model = myModelRepository.getModelByName("Gemma-3n-E2B-it-int4")

// 2. Initialize the LiteRT engine – no network required
LlmChatModelHelper.initialize(
    context = appContext,
    model = model,
    supportImage = true,
    supportAudio = false,
    onDone = { errMsg -> 
        if (errMsg.isNotEmpty()) Log.e("Demo", errMsg) 
    },
    systemInstruction = null,
    tools = emptyList(),
    enableConversationConstrainedDecoding = false,
    coroutineScope = null
)

// 3. Execute inference – all data remains in-process
LlmChatModelHelper.runInference(
    model = model,
    input = "What are the health benefits of green tea?",
    resultListener = { partial, done, _ ->
        if (!done) {
            println("Streaming: $partial")
        } else {
            println("Inference complete")
        }
    },
    cleanUpListener = { /* optional cleanup */ },
    onError = { err -> Log.e("Demo", err) },
    images = emptyList(),
    audioClips = emptyList(),
    coroutineScope = null,
    extraContext = null
)

This implementation confirms that once the model is downloaded, no additional network calls are performed during on-device inference, keeping all user interactions private.

Summary

• Offline-Only Execution: The google-ai-edge/gallery repository processes all LLM operations locally using the LiteRT-LM runtime, ensuring zero data transmission during inference.
• Sandboxed Storage: ModelManagerViewModel stores downloaded models in the app’s private externalFilesDir, preventing unauthorized access by other applications.
• In-Process Computation: The LlmChatModelHelper.kt class handles tokenization and tensor calculations within the native LiteRT library, which lacks network capabilities entirely.
• No Telemetry: User prompts, images, and audio are processed through conversation.sendMessageAsync() without cloud communication, as verified by the source code at lines 44‑51 and 79‑91.

Frequently Asked Questions

Does Google AI Edge Gallery send user data to external servers?

No. According to the source code in LlmChatModelHelper.kt, the sendMessageAsync() method processes all inputs locally within the device’s memory space. The only network activity occurs during the initial model download performed by ModelManagerViewModel, after which the application can operate entirely offline.

Where are the AI models stored on the device?

Model files are stored in the application’s private externalFilesDir directory, accessed via Android’s sandboxed storage APIs. This ensures the downloaded weights are isolated from other apps and remain encrypted at rest according to Android’s security model.

Can the app function without an internet connection?

Yes. After the initial model download is complete, the on-device inference pipeline requires no connectivity. The LiteRT runtime executes completely offline, allowing users to run prompts, analyze images, and process audio without any network access.

What runtime executes the inference operations?

The primary runtime is LiteRT-LM (formerly TensorFlow Lite), loaded through LlmChatModelHelper.initialize(). For compatible models, the app may alternatively use the MediaPipe LLM Inference API. Both runtimes execute natively on the device’s CPU, GPU, or NPU without network dependencies.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}