deep-dive

How Agent Skills Extend LLMs with External Integrations in Google AI Edge Gallery

April 6, 2026 google-ai-edge/gallery ↗

Agent Skills turn on-device Large Language Models into extensible assistants by injecting declarative metadata into system prompts and routing tool calls to either sandboxed JavaScript runtimes or native Android intents, enabling secure API integrations without cloud dependencies.

The Google AI Edge Gallery demonstrates how Agent Skills extend LLMs with external integrations through a declarative architecture that keeps inference local while allowing dynamic capability expansion. By parsing SKILL.md files at startup and appending skill metadata to the model's system prompt, the app enables the LLM to self-select and invoke external tools running in isolated execution environments.

Architecture Overview: Metadata Injection and Tool Selection

At the core of the system, each skill ships with a SKILL.md file containing a front-matter block with name, description, and optional metadata fields, followed by step-by-step instructions for the model. When a user opens the Agent Skills use-case, the Gallery app appends all skill metadata to the LLM's system prompt as described in the repository-wide README.md and detailed in skills/README.md.

The model then evaluates incoming requests against these descriptions. When a query matches a skill's description, the LLM generates a tool call—either run_js for JavaScript skills or run_intent for native operations—passing structured JSON data that triggers the corresponding execution path.

Execution Paths: JavaScript Runtime vs. Native Intents

The Gallery supports two distinct execution models for Agent Skills, defined in skills/README.md:

  • JavaScript Skills execute inside a hidden WebView that loads an HTML page, enabling sandboxed API calls, WebAssembly, and CDN resources.
  • Native App Intents map skills to Android or iOS intents for deep device integration like sending emails or SMS messages.

JavaScript Skills: The Hidden WebView Runtime

For JavaScript-based integrations, the app loads scripts/index.html within a hidden WebView. The page must expose an async function named window['ai_edge_gallery_get_result'](data, secret?) that accepts a JSON string from the LLM and returns a JSON string containing the execution result.

As shown in skills/built-in/calculate-hash/scripts/index.html, the runtime supports standard web APIs including fetch, allowing skills to query external services. The function must return an object containing one of the following keys:

  • result: Plain text displayed in the chat interface
  • image: Base64-encoded image data for inline rendering
  • webview: Object with url and aspectRatio for interactive embeds
  • error: Error message string if execution fails

Native Intent Skills: Deep Device Integration

Skills requiring device-level functionality use the run_intent tool handled by IntentHandler.kt in Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/agentchat/IntentHandler.kt. The handler parses JSON payloads using Moshi, constructs Android Intent objects (such as ACTION_SEND for email or ACTION_SENDTO for SMS), and launches the activity with context.startActivity(intent).

This path requires no JavaScript runtime and directly interfaces with the Android operating system, enabling seamless integration with system applications.

Building a JavaScript Skill: External API Integration

Creating a JavaScript skill requires two files: a SKILL.md descriptor and an HTML entry point. Consider a weather API integration example based on the repository structure:

SKILL.md declares the tool interface:

---
name: weather-reporter
description: Calls a third-party weather API and returns the current temperature.
metadata:
  require-secret: true
  require-secret-description: "Enter your OpenWeather API key"
---

## Instructions

Call the `run_js` tool with:
- script name: index.html
- data: JSON with field `city`: String

scripts/index.html implements the logic:

<!doctype html>
<html lang="en">
  <body>
    <script>
      window['ai_edge_gallery_get_result'] = async (data, secret) => {
        try {
          const { city } = JSON.parse(data);
          const apiKey = secret;
          const resp = await fetch(
            `https://api.openweathermap.org/data/2.5/weather?q=${encodeURIComponent(city)}&units=metric&appid=${apiKey}`
          );
          const json = await resp.json();
          
          if (!resp.ok) throw new Error(json.message);
          const result = `The current temperature in ${city} is ${json.main.temp}°C.`;
          return JSON.stringify({ result });
        } catch (e) {
          return JSON.stringify({ error: e.message });
        }
      };
    </script>
  </body>
</html>

When the user asks "What's the temperature in Paris?", the LLM identifies the matching skill description, triggers run_js, and the app supplies the parsed JSON data and user-provided API key to the WebView function.

Implementing Native Intent Skills: Email and SMS Workflows

Native skills follow a similar declaration pattern but target system applications. The SKILL.md instructs the LLM to call run_intent with specific parameters:

---
name: send-email
description: Send an email from the device.
---

## Instructions

Call the `run_intent` tool with:
- intent: send_email
- parameters: JSON with fields:
  - extra_email: String (recipient)
  - extra_subject: String (subject)
  - extra_text: String (body)

The IntentHandler.kt source contains the mapping logic for send_email, parsing the JSON payload and constructing the appropriate Android Intent with ACTION_SEND to open the email client pre-populated with the provided fields.

Security Model: Secret Management for API Keys

For JavaScript skills requiring authentication, the metadata block supports require-secret: true. When present, the Gallery app displays a native dialog to collect sensitive information (such as API keys) before execution. The secret passes as the second argument to ai_edge_gallery_get_result, ensuring credentials never appear in chat history or static skill configuration files.

This approach keeps secrets out of the declarative SKILL.md definitions while allowing dynamic credential injection at runtime. The calculate-hash and query-wikipedia built-in skills demonstrate this pattern for services requiring authentication tokens.

Result Handling: Text, Images, and Interactive WebViews

The JSON returned from JavaScript skills supports multiple output formats handled by the mobile UI:

  • Text responses: The result field displays as standard chat messages
  • Image rendering: Base64-encoded strings in image.base64 decode and display inline within the conversation
  • Interactive webviews: The webview object with url and aspectRatio renders as an embedded component inside the chat bubble, enabling rich interactive experiences like virtual pianos or maps

For example, an interactive map skill can return both a text description and a webview pointing to assets/webview.html, as demonstrated in the built-in interactive map skill structure.

Summary

Agent Skills extend LLMs with external integrations through a clean separation of concerns:

  • Metadata Layer: SKILL.md files provide declarative descriptions that augment the LLM's system prompt, enabling the model to self-select appropriate tools
  • Execution Layer: JavaScript skills run in sandboxed WebViews via window['ai_edge_gallery_get_result'], while native skills leverage IntentHandler.kt for device integration
  • Security Layer: Optional secret injection through require-secret: true keeps API keys out of static configuration and chat logs
  • Presentation Layer: Standardized JSON returns support text, images, and interactive webviews for flexible output rendering

This architecture enables developers to add capabilities to on-device models without modifying core application code or compromising local inference patterns.

Frequently Asked Questions

How does the LLM know when to invoke a specific skill?

The Gallery app appends all skill metadata (name and description from SKILL.md front-matter) to the LLM's system prompt at runtime according to the implementation in README.md. When processing user queries, the model uses this context to determine if a request matches a skill's described capabilities and generates the appropriate run_js or run_intent tool call with structured JSON parameters.

Can skills access the internet or only local device functions?

Skills support both approaches depending on the execution path. JavaScript skills execute in a hidden WebView with full access to fetch, WebAssembly, and CDN resources for internet connectivity, as documented in skills/README.md. Native intent skills trigger local Android activities like email composition or SMS without requiring network access, handled entirely within IntentHandler.kt.

Where are API keys and secrets stored when using external services?

Secrets are collected at runtime through native dialogs when a skill declares require-secret: true in its metadata block. The Gallery app passes these credentials as the second argument to ai_edge_gallery_get_result, ensuring keys never reside in SKILL.md files or chat logs, and only persist in the app's secure storage according to the secret management specification.

What file structure is required to create a new skill?

Every skill requires a directory containing a SKILL.md file with YAML front-matter (name, description, optional metadata) and markdown instructions. JavaScript skills additionally require a scripts/index.html file exposing the ai_edge_gallery_get_result function, while native skills rely on existing handlers in Android/src/app/src/main/java/com/google/ai/edge/gallery/customtasks/agentchat/IntentHandler.kt for supported intents like send_email or send_sms.

Have a question about this repo?

These articles cover the highlights, but your codebase questions are specific. Give your agent direct access to the source. Share this with your agent to get started:

Share the following with your agent to get started:
curl -s "https://instagit.com/install.md"
Works with
Claude Codex Cursor VS Code OpenClaw Any MCP Client

Maintain an open-source project? Get it listed too →