# How to Add Tool Definitions for Mobile Actions in Android > Learn how to add tool definitions for Mobile Actions in Android by annotating Kotlin functions and implementing execution logic. Follow our guide for seamless integration. - Repository: [google-ai-edge/gallery](https://github.com/google-ai-edge/gallery) - Tags: how-to-guide - Published: 2026-04-06 --- **To add tool definitions for Mobile Actions, annotate a Kotlin function with `@Tool` and `@ToolParam` in [`MobileActionsTools.kt`](https://github.com/google-ai-edge/gallery/blob/main/MobileActionsTools.kt), define a corresponding sealed-class `Action` in [`MobileActionsViewModel.kt`](https://github.com/google-ai-edge/gallery/blob/main/MobileActionsViewModel.kt), and implement the execution logic in the view-model's handler.** Mobile Actions is the on-device function-calling framework in the [google-ai-edge/gallery](https://github.com/google-ai-edge/gallery) repository that enables finetuned FunctionGemma models to invoke Android capabilities through **tool definitions**. Adding new tools requires implementing the `ToolSet` interface pattern used by the LLM runtime to discover and execute Kotlin methods as callable functions. The architecture bridges model-generated tool requests to concrete Android operations via type-safe callbacks. ## Understanding the Mobile Actions Architecture Before adding new tools, you need to understand how the system routes LLM requests to Android APIs. ### The Tool Execution Pipeline The Mobile Actions framework in `google-ai-edge/gallery` follows a four-stage pipeline: 1. **Discovery**: The `MobileActionsTask` registers `MobileActionsTools` (which implements `ToolSet`) with the LLM runtime, exposing annotated methods. 2. **Invocation**: When the model decides to call a tool, it serializes parameters and invokes the corresponding Kotlin method. 3. **Bridging**: The tool method creates an `Action` object (sealed class) and calls `onFunctionCalled()`, forwarding the request to the view-model. 4. **Execution**: `MobileActionsViewModel` handles the `Action`, performs the Android operation (e.g., CameraManager, Intent), and returns results to the LLM via `Map`. ### Key Source Files | File | Purpose | Path | |------|---------|------| | **MobileActionsTools.kt** | Declares annotated tools and forwards calls to typed `Action` objects. | [`Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/MobileActionsTools.kt`](https://github.com/google-ai-edge/gallery/blob/main/Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/MobileActionsTools.kt) | | **MobileActionsViewModel.kt** | Receives `Action` callbacks and implements actual Android logic. | [`Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/MobileActionsViewModel.kt`](https://github.com/google-ai-edge/gallery/blob/main/Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/MobileActionsViewModel.kt) | | **MobileActionsTask.kt** | Registers the `ToolSet` with the LLM runtime. | [`Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/MobileActionsTask.kt`](https://github.com/google-ai-edge/gallery/blob/main/Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/MobileActionsTask.kt) | | **MobileActionsModule.kt** | Hilt DI module providing the task graph. | [`Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/MobileActionsModule.kt`](https://github.com/google-ai-edge/gallery/blob/main/Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/MobileActionsModule.kt) | | **Function_Calling_Guide.md** | Documentation on annotation contracts and callback patterns. | [`Function_Calling_Guide.md`](https://github.com/google-ai-edge/gallery/blob/main/Function_Calling_Guide.md) | ## How to Add a New Tool Definition Follow this pattern to extend the tool set with new Android capabilities. ### Step 1: Annotate the Tool Function In [`MobileActionsTools.kt`](https://github.com/google-ai-edge/gallery/blob/main/MobileActionsTools.kt), add a public method annotated with **`@Tool`** and **`@ToolParam`**. The method name becomes the tool's identifier, and it must return a `Map` for the LLM response. ```kotlin // File: MobileActionsTools.kt /** * Plays a bundled sound file. */ @Tool(description = "Plays a sound file bundled in the app.") fun playSound( @ToolParam(description = "The name of the sound asset (without extension).") soundName: String, ): Map { Log.d(TAG, "Play sound: $soundName") // Forward to view-model via callback onFunctionCalled(PlaySoundAction(soundName = soundName)) // Return result to LLM return mapOf("result" to "success", "sound_name" to soundName) } ``` ### Step 2: Define the Action Data Class In [`MobileActionsViewModel.kt`](https://github.com/google-ai-edge/gallery/blob/main/MobileActionsViewModel.kt), extend the sealed `Action` hierarchy to represent this operation. ```kotlin // File: MobileActionsViewModel.kt (add to sealed class) data class PlaySoundAction(val soundName: String) : Action() ``` ### Step 3: Handle the Action Add the execution logic to the view-model's action handler, typically in a `when` expression processing incoming `Action` objects. ```kotlin // File: MobileActionsViewModel.kt (inside handleAction) is PlaySoundAction -> { val assetManager = appContext.assets val afd = assetManager.openFd("sounds/${action.soundName}.mp3") MediaPlayer().apply { setDataSource(afd.fileDescriptor, afd.startOffset, afd.length) prepare() start() } } ``` ### Step 4: Rebuild and Verify After compiling, the LLM runtime automatically discovers the new tool through reflection on `MobileActionsTools`. The FunctionGemma model can now generate calls to `playSound` with the specified `soundName` parameter. ## Complete Working Example Here is the full implementation flow for adding a "Play Sound" tool definition across the key files: **MobileActionsTools.kt** (Tool Definition): ```kotlin class MobileActionsTools @Inject constructor( private val onFunctionCalled: (Action) -> Unit ) : ToolSet { @Tool(description = "Plays a sound file bundled in the app.") fun playSound( @ToolParam(description = "The name of the sound asset (without extension).") soundName: String, ): Map { onFunctionCalled(PlaySoundAction(soundName)) return mapOf("status" to "playing", "asset" to soundName) } } ``` **MobileActionsViewModel.kt** (Action Definition and Handler): ```kotlin // Sealed class definition sealed class Action data class PlaySoundAction(val soundName: String) : Action() class MobileActionsViewModel @Inject constructor( private val appContext: Context ) : ViewModel() { fun handleAction(action: Action) { when (action) { is PlaySoundAction -> { val afd = appContext.assets.openFd("sounds/${action.soundName}.mp3") MediaPlayer().apply { setDataSource(afd.fileDescriptor, afd.startOffset, afd.length) prepare() start() } } // Handle other actions... } } } ``` **MobileActionsTask.kt** (Registration - no changes needed): ```kotlin class MobileActionsTask @Inject constructor( mobileActionsTools: MobileActionsTools ) { // Registers the ToolSet with the LLM runtime val toolSet: ToolSet = mobileActionsTools } ``` ## Summary - **Tool definitions** require `@Tool` and `@ToolParam` annotations in [`MobileActionsTools.kt`](https://github.com/google-ai-edge/gallery/blob/main/MobileActionsTools.kt) to expose Kotlin methods to the FunctionGemma model. - **Action objects** bridge the tool definition to Android implementation using a sealed class hierarchy defined in [`MobileActionsViewModel.kt`](https://github.com/google-ai-edge/gallery/blob/main/MobileActionsViewModel.kt). - **Return values** must be `Map` to provide structured results back to the LLM conversation. - **Registration** happens automatically through `MobileActionsTask` which supplies the `ToolSet` to the LLM runtime via dependency injection in [`MobileActionsModule.kt`](https://github.com/google-ai-edge/gallery/blob/main/MobileActionsModule.kt). ## Frequently Asked Questions ### What is the purpose of the `onFunctionCalled` callback? The `onFunctionCalled` callback bridges the tool definition to the view-model. When a tool method executes, it creates an `Action` object and passes it to this callback, which forwards the request to `MobileActionsViewModel`. According to the source code in [`MobileActionsTools.kt`](https://github.com/google-ai-edge/gallery/blob/main/MobileActionsTools.kt), this decouples the LLM-facing tool interface from the Android implementation details. ### How do I return structured data from a tool definition? Tool methods must return a `Map` that the LLM includes in its response. For example, returning `mapOf("result" to "success")` provides the model with confirmation that the action completed, which it can reference in subsequent conversation turns. This return value is defined in the method signature and populated before calling `onFunctionCalled`. ### Where should the actual Android API calls be implemented? Implement Android-specific logic in [`MobileActionsViewModel.kt`](https://github.com/google-ai-edge/gallery/blob/main/MobileActionsViewModel.kt) inside the action handler (typically a `when` expression). This view-model receives `Action` objects from the tool layer and performs operations like accessing `CameraManager`, sending Intents, or playing media. Keeping implementation details in the view-model ensures [`MobileActionsTools.kt`](https://github.com/google-ai-edge/gallery/blob/main/MobileActionsTools.kt) remains focused on parameter marshaling and LLM communication. ### Do I need to modify MobileActionsTask.kt to add new tools? No. [`MobileActionsTask.kt`](https://github.com/google-ai-edge/gallery/blob/main/MobileActionsTask.kt) registers the entire `ToolSet` with the LLM runtime once at initialization. New tools added to [`MobileActionsTools.kt`](https://github.com/google-ai-edge/gallery/blob/main/MobileActionsTools.kt) are automatically discovered through reflection when the app rebuilds, requiring no changes to the task registration code or the Hilt module in [`MobileActionsModule.kt`](https://github.com/google-ai-edge/gallery/blob/main/MobileActionsModule.kt).