# How to Define New Action Types for Mobile Actions in Google AI Edge Gallery

> Learn how to define new action types for Mobile Actions in the Google AI Edge Gallery. Extend ActionType, expose a matching Tool method, and implement runtime logic.

- Repository: [google-ai-edge/gallery](https://github.com/google-ai-edge/gallery)
- Tags: how-to-guide
- Published: 2026-04-06

---

**To define new action types for Mobile Actions, extend the `ActionType` enum in [`Actions.kt`](https://github.com/google-ai-edge/gallery/blob/main/Actions.kt), expose a matching `@Tool` method in [`MobileActionsTools.kt`](https://github.com/google-ai-edge/gallery/blob/main/MobileActionsTools.kt), and implement the runtime logic in `performAction` within [`MobileActionsViewModel.kt`](https://github.com/google-ai-edge/gallery/blob/main/MobileActionsViewModel.kt).**

The Google AI Edge Gallery demonstrates on-device LLM capabilities through Mobile Actions, a framework that translates natural language commands into Android system function calls. When you define new action types for Mobile Actions, you enable the LLM to trigger custom device behaviors ranging from camera capture to system settings adjustments. This requires modifying three tightly-coupled architectural components in the Kotlin source code to maintain type safety across the model-tool-executor pipeline.

## Understanding the Mobile Actions Architecture

Mobile Actions rely on a three-layer architecture that bridges the LLM's function calling with Android runtime operations:

| Component | Responsibility | Key File |
|-----------|----------------|----------|
| **Action Model** | Enumeration of supported types and sealed hierarchy of concrete actions | [`Actions.kt`](https://github.com/google-ai-edge/gallery/blob/main/Actions.kt) |
| **Tool Set** | Litert-LM `@Tool` methods exposing actions to the LLM | [`MobileActionsTools.kt`](https://github.com/google-ai-edge/gallery/blob/main/MobileActionsTools.kt) |
| **Executor** | `performAction` implementation routing actions to Android APIs | [`MobileActionsViewModel.kt`](https://github.com/google-ai-edge/gallery/blob/main/MobileActionsViewModel.kt) |

The **Action Model** defines what the UI can display and what parameters the model accepts. The **Tool Set** exposes these actions to the LLM as function calls via annotations. The **Executor** receives concrete `Action` instances from the callback and executes Android-specific implementation logic.

## Step-by-Step Implementation

Follow this sequence to add a custom action type that the LLM can invoke.

### Step 1: Extend the Action Model in Actions.kt

In [`Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/Actions.kt`](https://github.com/google-ai-edge/gallery/blob/main/Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/Actions.kt), declare a new enum constant and create a concrete subclass of `Action`.

```kotlin
enum class ActionType {
    /* existing entries … */
    ACTION_TAKE_PHOTO,           // ← new action type
}

/**
 * Action to capture a photo with the device camera.
 *
 * @param quality JPEG quality (0‑100). Optional – defaults to 80.
 */
class TakePhotoAction(
    val quality: Int = 80,
) : Action(
    type = ActionType.ACTION_TAKE_PHOTO,
    icon = Icons.Outlined.Camera,               // Choose an appropriate Material icon
    functionCallDetails = FunctionCallDetails(
        functionName = "takePhoto",
        parameters = listOf(Pair("quality", quality.toString()))
    )
)

```

**Key implementation details:**
- The enum constant `ACTION_TAKE_PHOTO` informs the UI which icons and labels to render.
- The `functionName` in `FunctionCallDetails` must exactly match the method name you'll expose in the tool set.
- The `icon` parameter uses Material Design icons for UI representation.

### Step 2: Add a Litert-LM Tool in MobileActionsTools.kt

In [`Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/MobileActionsTools.kt`](https://github.com/google-ai-edge/gallery/blob/main/Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/MobileActionsTools.kt), add a `@Tool` method that constructs your action subclass and forwards it to the view-model callback.

```kotlin
class MobileActionsTools(val onFunctionCalled: (Action) -> Unit) : ToolSet {

    // …existing tools…

    /** Captures a photo using the device camera. */
    @Tool(description = "Takes a photo with the device camera.")
    fun takePhoto(
        @ToolParam(description = "JPEG quality (0‑100). Optional; defaults to 80.") quality: Int = 80
    ): Map<String, String> {
        Log.d(TAG, "takePhoto called with quality=$quality")
        onFunctionCalled(TakePhotoAction(quality = quality))
        return mapOf("result" to "queued")
    }
}

```

**Critical requirements:**
- The method name `takePhoto` must match the `functionName` defined in the action class.
- `@ToolParam` annotations provide the LLM with human-readable parameter descriptions.
- The method invokes `onFunctionCalled` with a concrete `TakePhotoAction` instance, then returns a status map indicating successful queuing.

### Step 3: Implement the Executor Logic in MobileActionsViewModel.kt

In [`Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/MobileActionsViewModel.kt`](https://github.com/google-ai-edge/gallery/blob/main/Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/MobileActionsViewModel.kt), extend `performAction` to handle your new action type.

```kotlin
fun performAction(action: Action, context: Context): String {
    return when (action) {
        // …existing cases…

        // New case for taking a photo.
        is TakePhotoAction -> takePhoto(context, action.quality)

        else -> ""
    }
}

/** Starts the camera Intent and returns an error string if something goes wrong. */
private fun takePhoto(context: Context, quality: Int): String {
    return try {
        val intent = Intent(MediaStore.ACTION_IMAGE_CAPTURE).apply {
            putExtra("android.intent.extra.quickCapture", true)
            // Optional: pass quality via extras or store it for later processing.
        }
        context.startActivity(intent)
        ""
    } catch (e: Exception) {
        Log.e(TAG, "Failed to start camera", e)
        e.message ?: context.getString(R.string.unknown_error)
    }
}

```

**Execution flow:**
- The `when` expression detects `TakePhotoAction` and extracts the `quality` parameter.
- The implementation launches `MediaStore.ACTION_IMAGE_CAPTURE` via an Android `Intent`.
- Return an empty string on success; any non-empty string displays as a snackbar error in the UI.

### Step 4: Surface the Action in MobileActionsScreen.kt (Optional)

To make your new action discoverable in the sample prompt list, update [`Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/MobileActionsScreen.kt`](https://github.com/google-ai-edge/gallery/blob/main/Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/MobileActionsScreen.kt).

```kotlin
// Prompt templates – add a quick‑tap example.
PromptTemplate(
    labelResId = R.string.prompt_template_label_take_photo,
    prompt = "Take a photo with 90% quality"
)

// Sample action items – used for the icon list under the welcome message.
SampleActionItem(
    labelResId = R.string.prompt_template_label_take_photo,
    icon = Icons.Outlined.Camera
)

```

Add corresponding string resources in [`res/values/strings.xml`](https://github.com/google-ai-edge/gallery/blob/main/res/values/strings.xml) to complete the UI integration.

## Key Files Reference

| File | Path |
|------|------|
| **Actions.kt** | [`Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/Actions.kt`](https://github.com/google-ai-edge/gallery/blob/main/Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/Actions.kt) |
| **MobileActionsTools.kt** | [`Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/MobileActionsTools.kt`](https://github.com/google-ai-edge/gallery/blob/main/Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/MobileActionsTools.kt) |
| **MobileActionsViewModel.kt** | [`Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/MobileActionsViewModel.kt`](https://github.com/google-ai-edge/gallery/blob/main/Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/MobileActionsViewModel.kt) |
| **MobileActionsScreen.kt** | [`Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/MobileActionsScreen.kt`](https://github.com/google-ai-edge/gallery/blob/main/Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/mobileactions/MobileActionsScreen.kt) |
| **strings.xml** | [`res/values/strings.xml`](https://github.com/google-ai-edge/gallery/blob/main/res/values/strings.xml) |

## Summary

- **Model Layer**: Add enum constants to `ActionType` and subclass `Action` with `FunctionCallDetails` in [`Actions.kt`](https://github.com/google-ai-edge/gallery/blob/main/Actions.kt).
- **Tool Layer**: Expose methods using `@Tool` annotations in [`MobileActionsTools.kt`](https://github.com/google-ai-edge/gallery/blob/main/MobileActionsTools.kt), ensuring exact name matching with the action's `functionName`.
- **Executor Layer**: Extend `performAction` in [`MobileActionsViewModel.kt`](https://github.com/google-ai-edge/gallery/blob/main/MobileActionsViewModel.kt) with `when` branches that map actions to Android `Intent` calls.
- **UI Layer**: Update [`MobileActionsScreen.kt`](https://github.com/google-ai-edge/gallery/blob/main/MobileActionsScreen.kt) and [`strings.xml`](https://github.com/google-ai-edge/gallery/blob/main/strings.xml) to expose the capability in the sample prompts list.

## Frequently Asked Questions

### What is the relationship between ActionType and the @Tool method name?

The `ActionType` enum categorizes the action for UI rendering, while the `@Tool` method name (e.g., `takePhoto`) must exactly match the `functionName` parameter passed to `FunctionCallDetails` in your `Action` subclass. This string matching enables the Litert-LM framework to route LLM function calls to the correct tool method, which then constructs the concrete action instance.

### Can I define action types without modifying the UI?

Yes. Steps 1 through 3 are mandatory to define new action types for Mobile Actions and make them functional, but Step 4 is optional. If you skip [`MobileActionsScreen.kt`](https://github.com/google-ai-edge/gallery/blob/main/MobileActionsScreen.kt) updates, the action remains available for direct LLM invocation through natural language prompts, though it won't appear in the app's sample prompt templates.

### How does the LLM know which parameters to pass to custom actions?

The Litert-LM framework inspects `@ToolParam` annotations in [`MobileActionsTools.kt`](https://github.com/google-ai-edge/gallery/blob/main/MobileActionsTools.kt) to generate the function schema exposed to the LLM. The description strings in these annotations guide the model's understanding of expected values, ranges, and defaults, allowing it to populate the correct fields in the JSON function call payload.

### Where should permission handling be implemented for sensitive actions?

Implement Android permission checks within the executor methods in [`MobileActionsViewModel.kt`](https://github.com/google-ai-edge/gallery/blob/main/MobileActionsViewModel.kt) (e.g., inside `takePhoto()`), or handle them at the Activity level before invoking `performAction`. The `performAction` function returns error strings that the UI displays as snackbars, making it ideal for signaling permission denials back to the user.