# How Agent Skills Extend LLMs with External Integrations in Google AI Edge Gallery

> Learn how Agent Skills extend LLMs with external integrations securely on device. Inject metadata and route tool calls to JavaScript or Android for powerful, cloud-free assistants. Explore Google AI Edge Gallery.

- Repository: [google-ai-edge/gallery](https://github.com/google-ai-edge/gallery)
- Tags: deep-dive
- Published: 2026-04-06

---

**Agent Skills turn on-device Large Language Models into extensible assistants by injecting declarative metadata into system prompts and routing tool calls to either sandboxed JavaScript runtimes or native Android intents, enabling secure API integrations without cloud dependencies.**

The Google AI Edge Gallery demonstrates how Agent Skills extend LLMs with external integrations through a declarative architecture that keeps inference local while allowing dynamic capability expansion. By parsing [`SKILL.md`](https://github.com/google-ai-edge/gallery/blob/main/SKILL.md) files at startup and appending skill metadata to the model's system prompt, the app enables the LLM to self-select and invoke external tools running in isolated execution environments.

## Architecture Overview: Metadata Injection and Tool Selection

At the core of the system, each skill ships with a [`SKILL.md`](https://github.com/google-ai-edge/gallery/blob/main/SKILL.md) file containing a front-matter block with `name`, `description`, and optional `metadata` fields, followed by step-by-step instructions for the model. When a user opens the *Agent Skills* use-case, the Gallery app appends **all** skill metadata to the LLM's system prompt as described in the repository-wide [`README.md`](https://github.com/google-ai-edge/gallery/blob/main/README.md) and detailed in [`skills/README.md`](https://github.com/google-ai-edge/gallery/blob/main/skills/README.md).

The model then evaluates incoming requests against these descriptions. When a query matches a skill's description, the LLM generates a tool call—either `run_js` for JavaScript skills or `run_intent` for native operations—passing structured JSON data that triggers the corresponding execution path.

## Execution Paths: JavaScript Runtime vs. Native Intents

The Gallery supports two distinct execution models for Agent Skills, defined in [`skills/README.md`](https://github.com/google-ai-edge/gallery/blob/main/skills/README.md):

- **JavaScript Skills** execute inside a hidden WebView that loads an HTML page, enabling sandboxed API calls, WebAssembly, and CDN resources.
- **Native App Intents** map skills to Android or iOS intents for deep device integration like sending emails or SMS messages.

### JavaScript Skills: The Hidden WebView Runtime

For JavaScript-based integrations, the app loads [`scripts/index.html`](https://github.com/google-ai-edge/gallery/blob/main/scripts/index.html) within a hidden WebView. The page must expose an async function named `window['ai_edge_gallery_get_result'](data, secret?)` that accepts a JSON string from the LLM and returns a JSON string containing the execution result.

As shown in [`skills/built-in/calculate-hash/scripts/index.html`](https://github.com/google-ai-edge/gallery/blob/main/skills/built-in/calculate-hash/scripts/index.html), the runtime supports standard web APIs including `fetch`, allowing skills to query external services. The function must return an object containing one of the following keys:

- `result`: Plain text displayed in the chat interface
- `image`: Base64-encoded image data for inline rendering
- `webview`: Object with `url` and `aspectRatio` for interactive embeds
- `error`: Error message string if execution fails

### Native Intent Skills: Deep Device Integration

Skills requiring device-level functionality use the `run_intent` tool handled by [`IntentHandler.kt`](https://github.com/google-ai-edge/gallery/blob/main/IntentHandler.kt) in [`Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/agentchat/IntentHandler.kt`](https://github.com/google-ai-edge/gallery/blob/main/Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/agentchat/IntentHandler.kt). The handler parses JSON payloads using Moshi, constructs Android `Intent` objects (such as `ACTION_SEND` for email or `ACTION_SENDTO` for SMS), and launches the activity with `context.startActivity(intent)`.

This path requires no JavaScript runtime and directly interfaces with the Android operating system, enabling seamless integration with system applications.

## Building a JavaScript Skill: External API Integration

Creating a JavaScript skill requires two files: a [`SKILL.md`](https://github.com/google-ai-edge/gallery/blob/main/SKILL.md) descriptor and an HTML entry point. Consider a weather API integration example based on the repository structure:

**[`SKILL.md`](https://github.com/google-ai-edge/gallery/blob/main/SKILL.md)** declares the tool interface:

```markdown
---
name: weather-reporter
description: Calls a third-party weather API and returns the current temperature.
metadata:
  require-secret: true
  require-secret-description: "Enter your OpenWeather API key"
---

## Instructions

Call the `run_js` tool with:
- script name: index.html
- data: JSON with field `city`: String

```

**[`scripts/index.html`](https://github.com/google-ai-edge/gallery/blob/main/scripts/index.html)** implements the logic:

```html
<!doctype html>
<html lang="en">
  <body>
    <script>
      window['ai_edge_gallery_get_result'] = async (data, secret) => {
        try {
          const { city } = JSON.parse(data);
          const apiKey = secret;
          const resp = await fetch(
            `https://api.openweathermap.org/data/2.5/weather?q=${encodeURIComponent(city)}&units=metric&appid=${apiKey}`
          );
          const json = await resp.json();
          
          if (!resp.ok) throw new Error(json.message);
          const result = `The current temperature in ${city} is ${json.main.temp}°C.`;
          return JSON.stringify({ result });
        } catch (e) {
          return JSON.stringify({ error: e.message });
        }
      };
    </script>
  </body>
</html>

```

When the user asks "What's the temperature in Paris?", the LLM identifies the matching skill description, triggers `run_js`, and the app supplies the parsed JSON data and user-provided API key to the WebView function.

## Implementing Native Intent Skills: Email and SMS Workflows

Native skills follow a similar declaration pattern but target system applications. The [`SKILL.md`](https://github.com/google-ai-edge/gallery/blob/main/SKILL.md) instructs the LLM to call `run_intent` with specific parameters:

```markdown
---
name: send-email
description: Send an email from the device.
---

## Instructions

Call the `run_intent` tool with:
- intent: send_email
- parameters: JSON with fields:
  - extra_email: String (recipient)
  - extra_subject: String (subject)
  - extra_text: String (body)

```

The [`IntentHandler.kt`](https://github.com/google-ai-edge/gallery/blob/main/IntentHandler.kt) source contains the mapping logic for `send_email`, parsing the JSON payload and constructing the appropriate Android `Intent` with `ACTION_SEND` to open the email client pre-populated with the provided fields.

## Security Model: Secret Management for API Keys

For JavaScript skills requiring authentication, the `metadata` block supports `require-secret: true`. When present, the Gallery app displays a native dialog to collect sensitive information (such as API keys) before execution. The secret passes as the second argument to `ai_edge_gallery_get_result`, ensuring credentials never appear in chat history or static skill configuration files.

This approach keeps secrets out of the declarative [`SKILL.md`](https://github.com/google-ai-edge/gallery/blob/main/SKILL.md) definitions while allowing dynamic credential injection at runtime. The calculate-hash and query-wikipedia built-in skills demonstrate this pattern for services requiring authentication tokens.

## Result Handling: Text, Images, and Interactive WebViews

The JSON returned from JavaScript skills supports multiple output formats handled by the mobile UI:

- **Text responses**: The `result` field displays as standard chat messages
- **Image rendering**: Base64-encoded strings in `image.base64` decode and display inline within the conversation
- **Interactive webviews**: The `webview` object with `url` and `aspectRatio` renders as an embedded component inside the chat bubble, enabling rich interactive experiences like virtual pianos or maps

For example, an interactive map skill can return both a text description and a webview pointing to [`assets/webview.html`](https://github.com/google-ai-edge/gallery/blob/main/assets/webview.html), as demonstrated in the built-in interactive map skill structure.

## Summary

Agent Skills extend LLMs with external integrations through a clean separation of concerns:

- **Metadata Layer**: [`SKILL.md`](https://github.com/google-ai-edge/gallery/blob/main/SKILL.md) files provide declarative descriptions that augment the LLM's system prompt, enabling the model to self-select appropriate tools
- **Execution Layer**: JavaScript skills run in sandboxed WebViews via `window['ai_edge_gallery_get_result']`, while native skills leverage [`IntentHandler.kt`](https://github.com/google-ai-edge/gallery/blob/main/IntentHandler.kt) for device integration
- **Security Layer**: Optional secret injection through `require-secret: true` keeps API keys out of static configuration and chat logs
- **Presentation Layer**: Standardized JSON returns support text, images, and interactive webviews for flexible output rendering

This architecture enables developers to add capabilities to on-device models without modifying core application code or compromising local inference patterns.

## Frequently Asked Questions

### How does the LLM know when to invoke a specific skill?

The Gallery app appends all skill metadata (name and description from [`SKILL.md`](https://github.com/google-ai-edge/gallery/blob/main/SKILL.md) front-matter) to the LLM's system prompt at runtime according to the implementation in [`README.md`](https://github.com/google-ai-edge/gallery/blob/main/README.md). When processing user queries, the model uses this context to determine if a request matches a skill's described capabilities and generates the appropriate `run_js` or `run_intent` tool call with structured JSON parameters.

### Can skills access the internet or only local device functions?

Skills support both approaches depending on the execution path. JavaScript skills execute in a hidden WebView with full access to `fetch`, WebAssembly, and CDN resources for internet connectivity, as documented in [`skills/README.md`](https://github.com/google-ai-edge/gallery/blob/main/skills/README.md). Native intent skills trigger local Android activities like email composition or SMS without requiring network access, handled entirely within [`IntentHandler.kt`](https://github.com/google-ai-edge/gallery/blob/main/IntentHandler.kt).

### Where are API keys and secrets stored when using external services?

Secrets are collected at runtime through native dialogs when a skill declares `require-secret: true` in its `metadata` block. The Gallery app passes these credentials as the second argument to `ai_edge_gallery_get_result`, ensuring keys never reside in [`SKILL.md`](https://github.com/google-ai-edge/gallery/blob/main/SKILL.md) files or chat logs, and only persist in the app's secure storage according to the secret management specification.

### What file structure is required to create a new skill?

Every skill requires a directory containing a [`SKILL.md`](https://github.com/google-ai-edge/gallery/blob/main/SKILL.md) file with YAML front-matter (name, description, optional metadata) and markdown instructions. JavaScript skills additionally require a [`scripts/index.html`](https://github.com/google-ai-edge/gallery/blob/main/scripts/index.html) file exposing the `ai_edge_gallery_get_result` function, while native skills rely on existing handlers in [`Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/agentchat/IntentHandler.kt`](https://github.com/google-ai-edge/gallery/blob/main/Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/agentchat/IntentHandler.kt) for supported intents like `send_email` or `send_sms`.