How to Extend Mobile Actions with Custom Native Functionality: A Step-by-Step Guide

To extend Mobile Actions with custom native functionality, annotate a Kotlin method with @Tool in MobileActionsTools.kt, define a corresponding Action subclass in Actions.kt, and implement the native Android logic in MobileActionsViewModel.kt.

The Mobile Actions demo in the google-ai-edge/gallery repository demonstrates how Function Gemma models trigger native Android capabilities through function calling. Developers can extend Mobile Actions with custom native functionality—such as toggling Bluetooth, adjusting system settings, or accessing sensors—by following a consistent four-layer architecture. This guide walks through adding a new capability while preserving the existing inference flow.

Understanding the Mobile Actions Architecture

The Mobile Actions implementation consists of four distinct layers that handle the flow from model inference to native execution:

1. Tool definitions (MobileActionsTools.kt) – Kotlin functions annotated with @Tool that the LiteRT-LM inference engine exposes to the model.
2. Action model (Actions.kt) – Plain data classes that describe concrete native operations, including parameters and UI metadata.
3. View-model orchestration (MobileActionsViewModel.kt) – Receives Action objects and executes real Android APIs.
4. Task wiring and UI (MobileActionsTask.kt and MobileActionsScreen.kt) – Registers tools with the inference engine and displays results.

Step 1: Define the Tool with @Tool Annotation

Create a new method in MobileActionsTools.kt and annotate it with @Tool. The method must return a Map<String, String> that the model can read and invoke the injected onFunctionCalled callback with a new Action subclass.

/** Toggles Bluetooth on/off. */
@Tool(description = "Enables or disables Bluetooth on the device.")
fun setBluetooth(
    @ToolParam(description = "true to enable, false to disable") enable: Boolean
): Map<String, String> {
    Log.d(TAG, "setBluetooth enable=$enable")
    onFunctionCalled(SetBluetoothAction(enable))
    return mapOf("result" to "success", "enabled" to enable.toString())
}

The @Tool annotation marks this function as available for function calling, while @ToolParam documents the parameters for the model. When the model invokes setBluetooth, the method creates a SetBluetoothAction and passes it to the callback.

Step 2: Create the Action Model Class

Extend Actions.kt with a concrete Action subclass that holds parameters and UI metadata. You must also add a corresponding entry to the ActionType enum.

First, update the enum in Actions.kt:

enum class ActionType {
    ACTION_SET_FLASHLIGHT,
    ACTION_SET_WIFI,
    ACTION_OPEN_MAPS,
    ACTION_SET_BLUETOOTH,  // Add your new action type here
    // ... other actions
}

Then define the action class:

class SetBluetoothAction(val enable: Boolean) :
  Action(
    type = ActionType.ACTION_SET_BLUETOOTH,
    icon = Icons.Outlined.Bluetooth,
    functionCallDetails = FunctionCallDetails(
      functionName = "setBluetooth",
      parameters = listOf(Pair("enable", enable.toString()))
    )
  )

This class links the tool definition to the native implementation, providing the icon and function call details that appear in the UI.

Step 3: Implement Native Logic in MobileActionsViewModel

Add a handler in MobileActionsViewModel.kt inside the performAction method. This executes the actual Android Bluetooth APIs when the action is received.

Update the when expression in performAction:

// Toggle Bluetooth.
is SetBluetoothAction ->
  setBluetooth(context = context, enable = action.enable)

Then implement the helper method using BluetoothAdapter:

private fun setBluetooth(context: Context, enable: Boolean): String {
  val btAdapter = android.bluetooth.BluetoothAdapter.getDefaultAdapter()
  return try {
    if (enable) btAdapter?.enable() else btAdapter?.disable()
    ""
  } catch (e: Exception) {
    Log.e(TAG, "Failed to set Bluetooth", e)
    e.message ?: context.getString(R.string.unknown_error)
  }
}

The view-model bridges the Action objects from the model to native Android APIs, handling errors and returning status strings for the UI.

Step 4: Register the Tool (No-Code Registration)

In MobileActionsTask.kt, the tools list is initialized once and automatically exposes all @Tool methods:

private val tools = listOf(tool(MobileActionsTools(onFunctionCalled = { curActions.add(it) })))

Because MobileActionsTools is instantiated as a single component, adding a method to that class automatically includes it in the model's available function set. No additional registration code is required in the task file.

Step 5: Optional UI Customization

If your action requires a custom icon in the chat bubble, specify the icon in the Action constructor as shown in Step 2. The MobileActionsScreen.kt composable automatically renders Action.icon without requiring modifications to the UI code. You can use any Material Design icon from androidx.compose.material.icons or custom vector assets.

Testing Your Custom Functionality

After implementing the four steps above:

1. Ensure the Mobile Actions model (such as MobileActions-270M) is loaded in the demo app.
2. Build and deploy the Android application.
3. Prompt the model with a natural language request like "Turn Bluetooth on" or "Disable Bluetooth."
4. Verify that the model emits a function call, the tool creates a SetBluetoothAction, and the view-model executes the native Bluetooth logic.

Summary

Extending Mobile Actions with custom native functionality requires changes to three specific files in the google-ai-edge/gallery repository:

• MobileActionsTools.kt – Add @Tool annotated methods to expose functions to the model.
• Actions.kt – Create Action subclasses and update the ActionType enum to represent the operation.
• MobileActionsViewModel.kt – Handle the new Action type in performAction and implement native Android API calls.

The architecture automatically wires tools through MobileActionsTask.kt and renders UI through MobileActionsScreen.kt, allowing you to focus on the native implementation logic.

Frequently Asked Questions

What is the Mobile Actions demo?

The Mobile Actions demo is an Android application in the google-ai-edge/gallery repository that demonstrates function calling with Function Gemma models. It allows a small language model to control native device capabilities—such as the flashlight, Wi-Fi, and calendar—through Kotlin tools annotated with @Tool and executed via the LiteRT-LM inference engine.

Do I need to modify MobileActionsTask.kt for every new tool?

No. The MobileActionsTask.kt file instantiates MobileActionsTools as a single component and passes it to the inference engine. Since the tool() function introspects all @Tool annotated methods automatically, adding a new method to MobileActionsTools.kt exposes it to the model without requiring changes to the task registration code.

How does the model know which functions are available?

The LiteRT-LM inference engine uses reflection to scan the MobileActionsTools instance for methods marked with @Tool. It generates a function schema from the method signature and @ToolParam annotations, which is then provided to the model in the system prompt. When the model generates a function call, the engine maps it back to the corresponding Kotlin method.

Can I use any Android API with Mobile Actions?

Yes, provided the app has the necessary permissions. The MobileActionsViewModel.kt executes standard Android APIs—such as BluetoothAdapter, WifiManager, or Intent-based actions—when handling Action objects. You can extend Mobile Actions with any capability that can be triggered from a Kotlin function, including hardware sensors, system settings, or third-party app integrations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}