How to Add Tool Definitions for Mobile Actions in Android

To add tool definitions for Mobile Actions, annotate a Kotlin function with @Tool and @ToolParam in MobileActionsTools.kt, define a corresponding sealed-class Action in MobileActionsViewModel.kt, and implement the execution logic in the view-model's handler.

Mobile Actions is the on-device function-calling framework in the google-ai-edge/gallery repository that enables finetuned FunctionGemma models to invoke Android capabilities through tool definitions. Adding new tools requires implementing the ToolSet interface pattern used by the LLM runtime to discover and execute Kotlin methods as callable functions. The architecture bridges model-generated tool requests to concrete Android operations via type-safe callbacks.

Understanding the Mobile Actions Architecture

Before adding new tools, you need to understand how the system routes LLM requests to Android APIs.

The Tool Execution Pipeline

The Mobile Actions framework in google-ai-edge/gallery follows a four-stage pipeline:

1. Discovery: The MobileActionsTask registers MobileActionsTools (which implements ToolSet) with the LLM runtime, exposing annotated methods.
2. Invocation: When the model decides to call a tool, it serializes parameters and invokes the corresponding Kotlin method.
3. Bridging: The tool method creates an Action object (sealed class) and calls onFunctionCalled(), forwarding the request to the view-model.
4. Execution: MobileActionsViewModel handles the Action, performs the Android operation (e.g., CameraManager, Intent), and returns results to the LLM via Map<String, String>.

Key Source Files

File	Purpose	Path
MobileActionsTools.kt	Declares annotated tools and forwards calls to typed Action objects.	Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/MobileActionsTools.kt
MobileActionsViewModel.kt	Receives Action callbacks and implements actual Android logic.	Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/MobileActionsViewModel.kt
MobileActionsTask.kt	Registers the ToolSet with the LLM runtime.	Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/MobileActionsTask.kt
MobileActionsModule.kt	Hilt DI module providing the task graph.	Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/MobileActionsModule.kt
Function_Calling_Guide.md	Documentation on annotation contracts and callback patterns.	Function_Calling_Guide.md

How to Add a New Tool Definition

Follow this pattern to extend the tool set with new Android capabilities.

Step 1: Annotate the Tool Function

In MobileActionsTools.kt, add a public method annotated with @Tool and @ToolParam. The method name becomes the tool's identifier, and it must return a Map<String, String> for the LLM response.

// File: MobileActionsTools.kt
/**
 * Plays a bundled sound file.
 */
@Tool(description = "Plays a sound file bundled in the app.")
fun playSound(
    @ToolParam(description = "The name of the sound asset (without extension).") soundName: String,
): Map<String, String> {
    Log.d(TAG, "Play sound: $soundName")
    
    // Forward to view-model via callback
    onFunctionCalled(PlaySoundAction(soundName = soundName))
    
    // Return result to LLM
    return mapOf("result" to "success", "sound_name" to soundName)
}

Step 2: Define the Action Data Class

In MobileActionsViewModel.kt, extend the sealed Action hierarchy to represent this operation.

// File: MobileActionsViewModel.kt (add to sealed class)
data class PlaySoundAction(val soundName: String) : Action()

Step 3: Handle the Action

Add the execution logic to the view-model's action handler, typically in a when expression processing incoming Action objects.

// File: MobileActionsViewModel.kt (inside handleAction)
is PlaySoundAction -> {
    val assetManager = appContext.assets
    val afd = assetManager.openFd("sounds/${action.soundName}.mp3")
    
    MediaPlayer().apply {
        setDataSource(afd.fileDescriptor, afd.startOffset, afd.length)
        prepare()
        start()
    }
}

Step 4: Rebuild and Verify

After compiling, the LLM runtime automatically discovers the new tool through reflection on MobileActionsTools. The FunctionGemma model can now generate calls to playSound with the specified soundName parameter.

Complete Working Example

Here is the full implementation flow for adding a "Play Sound" tool definition across the key files:

MobileActionsTools.kt (Tool Definition):

class MobileActionsTools @Inject constructor(
    private val onFunctionCalled: (Action) -> Unit
) : ToolSet {
    
    @Tool(description = "Plays a sound file bundled in the app.")
    fun playSound(
        @ToolParam(description = "The name of the sound asset (without extension).") soundName: String,
    ): Map<String, String> {
        onFunctionCalled(PlaySoundAction(soundName))
        return mapOf("status" to "playing", "asset" to soundName)
    }
}

MobileActionsViewModel.kt (Action Definition and Handler):

// Sealed class definition
sealed class Action
data class PlaySoundAction(val soundName: String) : Action()

class MobileActionsViewModel @Inject constructor(
    private val appContext: Context
) : ViewModel() {
    
    fun handleAction(action: Action) {
        when (action) {
            is PlaySoundAction -> {
                val afd = appContext.assets.openFd("sounds/${action.soundName}.mp3")
                MediaPlayer().apply {
                    setDataSource(afd.fileDescriptor, afd.startOffset, afd.length)
                    prepare()
                    start()
                }
            }
            // Handle other actions...
        }
    }
}

MobileActionsTask.kt (Registration - no changes needed):

class MobileActionsTask @Inject constructor(
    mobileActionsTools: MobileActionsTools
) {
    // Registers the ToolSet with the LLM runtime
    val toolSet: ToolSet = mobileActionsTools
}

Summary

• Tool definitions require @Tool and @ToolParam annotations in MobileActionsTools.kt to expose Kotlin methods to the FunctionGemma model.
• Action objects bridge the tool definition to Android implementation using a sealed class hierarchy defined in MobileActionsViewModel.kt.
• Return values must be Map<String, String> to provide structured results back to the LLM conversation.
• Registration happens automatically through MobileActionsTask which supplies the ToolSet to the LLM runtime via dependency injection in MobileActionsModule.kt.

Frequently Asked Questions

What is the purpose of the onFunctionCalled callback?

The onFunctionCalled callback bridges the tool definition to the view-model. When a tool method executes, it creates an Action object and passes it to this callback, which forwards the request to MobileActionsViewModel. According to the source code in MobileActionsTools.kt, this decouples the LLM-facing tool interface from the Android implementation details.

How do I return structured data from a tool definition?

Tool methods must return a Map<String, String> that the LLM includes in its response. For example, returning mapOf("result" to "success") provides the model with confirmation that the action completed, which it can reference in subsequent conversation turns. This return value is defined in the method signature and populated before calling onFunctionCalled.

Where should the actual Android API calls be implemented?

Implement Android-specific logic in MobileActionsViewModel.kt inside the action handler (typically a when expression). This view-model receives Action objects from the tool layer and performs operations like accessing CameraManager, sending Intents, or playing media. Keeping implementation details in the view-model ensures MobileActionsTools.kt remains focused on parameter marshaling and LLM communication.

Do I need to modify MobileActionsTask.kt to add new tools?

No. MobileActionsTask.kt registers the entire ToolSet with the LLM runtime once at initialization. New tools added to MobileActionsTools.kt are automatically discovered through reflection when the app rebuilds, requiring no changes to the task registration code or the Hilt module in MobileActionsModule.kt.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}