How to Define New Action Types for Mobile Actions in Google AI Edge Gallery

To define new action types for Mobile Actions, extend the ActionType enum in Actions.kt, expose a matching @Tool method in MobileActionsTools.kt, and implement the runtime logic in performAction within MobileActionsViewModel.kt.

The Google AI Edge Gallery demonstrates on-device LLM capabilities through Mobile Actions, a framework that translates natural language commands into Android system function calls. When you define new action types for Mobile Actions, you enable the LLM to trigger custom device behaviors ranging from camera capture to system settings adjustments. This requires modifying three tightly-coupled architectural components in the Kotlin source code to maintain type safety across the model-tool-executor pipeline.

Understanding the Mobile Actions Architecture

Mobile Actions rely on a three-layer architecture that bridges the LLM's function calling with Android runtime operations:

Component	Responsibility	Key File
Action Model	Enumeration of supported types and sealed hierarchy of concrete actions	Actions.kt
Tool Set	Litert-LM @Tool methods exposing actions to the LLM	MobileActionsTools.kt
Executor	performAction implementation routing actions to Android APIs	MobileActionsViewModel.kt

The Action Model defines what the UI can display and what parameters the model accepts. The Tool Set exposes these actions to the LLM as function calls via annotations. The Executor receives concrete Action instances from the callback and executes Android-specific implementation logic.

Step-by-Step Implementation

Follow this sequence to add a custom action type that the LLM can invoke.

Step 1: Extend the Action Model in Actions.kt

In Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/Actions.kt, declare a new enum constant and create a concrete subclass of Action.

enum class ActionType {
    /* existing entries … */
    ACTION_TAKE_PHOTO,           // ← new action type
}

/**
 * Action to capture a photo with the device camera.
 *
 * @param quality JPEG quality (0‑100). Optional – defaults to 80.
 */
class TakePhotoAction(
    val quality: Int = 80,
) : Action(
    type = ActionType.ACTION_TAKE_PHOTO,
    icon = Icons.Outlined.Camera,               // Choose an appropriate Material icon
    functionCallDetails = FunctionCallDetails(
        functionName = "takePhoto",
        parameters = listOf(Pair("quality", quality.toString()))
    )
)

Key implementation details:

• The enum constant ACTION_TAKE_PHOTO informs the UI which icons and labels to render.
• The functionName in FunctionCallDetails must exactly match the method name you'll expose in the tool set.
• The icon parameter uses Material Design icons for UI representation.

Step 2: Add a Litert-LM Tool in MobileActionsTools.kt

In Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/MobileActionsTools.kt, add a @Tool method that constructs your action subclass and forwards it to the view-model callback.

class MobileActionsTools(val onFunctionCalled: (Action) -> Unit) : ToolSet {

    // …existing tools…

    /** Captures a photo using the device camera. */
    @Tool(description = "Takes a photo with the device camera.")
    fun takePhoto(
        @ToolParam(description = "JPEG quality (0‑100). Optional; defaults to 80.") quality: Int = 80
    ): Map<String, String> {
        Log.d(TAG, "takePhoto called with quality=$quality")
        onFunctionCalled(TakePhotoAction(quality = quality))
        return mapOf("result" to "queued")
    }
}

Critical requirements:

• The method name takePhoto must match the functionName defined in the action class.
• @ToolParam annotations provide the LLM with human-readable parameter descriptions.
• The method invokes onFunctionCalled with a concrete TakePhotoAction instance, then returns a status map indicating successful queuing.

Step 3: Implement the Executor Logic in MobileActionsViewModel.kt

In Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/MobileActionsViewModel.kt, extend performAction to handle your new action type.

fun performAction(action: Action, context: Context): String {
    return when (action) {
        // …existing cases…

        // New case for taking a photo.
        is TakePhotoAction -> takePhoto(context, action.quality)

        else -> ""
    }
}

/** Starts the camera Intent and returns an error string if something goes wrong. */
private fun takePhoto(context: Context, quality: Int): String {
    return try {
        val intent = Intent(MediaStore.ACTION_IMAGE_CAPTURE).apply {
            putExtra("android.intent.extra.quickCapture", true)
            // Optional: pass quality via extras or store it for later processing.
        }
        context.startActivity(intent)
        ""
    } catch (e: Exception) {
        Log.e(TAG, "Failed to start camera", e)
        e.message ?: context.getString(R.string.unknown_error)
    }
}

Execution flow:

• The when expression detects TakePhotoAction and extracts the quality parameter.
• The implementation launches MediaStore.ACTION_IMAGE_CAPTURE via an Android Intent.
• Return an empty string on success; any non-empty string displays as a snackbar error in the UI.

Step 4: Surface the Action in MobileActionsScreen.kt (Optional)

To make your new action discoverable in the sample prompt list, update Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/MobileActionsScreen.kt.

// Prompt templates – add a quick‑tap example.
PromptTemplate(
    labelResId = R.string.prompt_template_label_take_photo,
    prompt = "Take a photo with 90% quality"
)

// Sample action items – used for the icon list under the welcome message.
SampleActionItem(
    labelResId = R.string.prompt_template_label_take_photo,
    icon = Icons.Outlined.Camera
)

Add corresponding string resources in res/values/strings.xml to complete the UI integration.

Key Files Reference

File	Path
Actions.kt	Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/Actions.kt
MobileActionsTools.kt	Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/MobileActionsTools.kt
MobileActionsViewModel.kt	Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/MobileActionsViewModel.kt
MobileActionsScreen.kt	Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/MobileActionsScreen.kt
strings.xml	res/values/strings.xml

Summary

• Model Layer: Add enum constants to ActionType and subclass Action with FunctionCallDetails in Actions.kt.
• Tool Layer: Expose methods using @Tool annotations in MobileActionsTools.kt, ensuring exact name matching with the action's functionName.
• Executor Layer: Extend performAction in MobileActionsViewModel.kt with when branches that map actions to Android Intent calls.
• UI Layer: Update MobileActionsScreen.kt and strings.xml to expose the capability in the sample prompts list.

Frequently Asked Questions

What is the relationship between ActionType and the @Tool method name?

The ActionType enum categorizes the action for UI rendering, while the @Tool method name (e.g., takePhoto) must exactly match the functionName parameter passed to FunctionCallDetails in your Action subclass. This string matching enables the Litert-LM framework to route LLM function calls to the correct tool method, which then constructs the concrete action instance.

Can I define action types without modifying the UI?

Yes. Steps 1 through 3 are mandatory to define new action types for Mobile Actions and make them functional, but Step 4 is optional. If you skip MobileActionsScreen.kt updates, the action remains available for direct LLM invocation through natural language prompts, though it won't appear in the app's sample prompt templates.

How does the LLM know which parameters to pass to custom actions?

The Litert-LM framework inspects @ToolParam annotations in MobileActionsTools.kt to generate the function schema exposed to the LLM. The description strings in these annotations guide the model's understanding of expected values, ranges, and defaults, allowing it to populate the correct fields in the JSON function call payload.

Where should permission handling be implemented for sensitive actions?

Implement Android permission checks within the executor methods in MobileActionsViewModel.kt (e.g., inside takePhoto()), or handle them at the Activity level before invoking performAction. The performAction function returns error strings that the UI displays as snackbars, making it ideal for signaling permission denials back to the user.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}