Evaluating Agents on Mobile App Automation with Android World: A Complete Framework

The bojieli/ai-agent-book repository provides a modular framework for benchmarking autonomous AI agents on 116 real-world Android tasks using controlled experiments with configurable observation pipelines and reproducible metrics.

Evaluating agents on mobile app automation with Android World requires a standardized environment that can simulate realistic device interactions while measuring LLM performance accurately. The ai-agent-book repository implements exactly this through its Chapter 6 Android World benchmark, which integrates tightly with the AgentWorld simulation layer to provide deterministic testing across settings management, file operations, and third-party app interactions.

Android World Benchmark Architecture

The Android World implementation in chapter6/android-world/ provides a comprehensive suite for testing agent capabilities on actual Android devices through a Docker-based emulator environment.

Core Library and Provider Abstraction

The agentbook/ directory contains the unified provider interface that abstracts LLM backends through a central registry pattern. Key files include agentbook/providers/registry.py, which maps provider names to concrete implementations, along with openrouter.py, models.py, legacy.py, and resolution.py for handling request routing and model selection. This abstraction allows seamless switching between OpenRouter and legacy model APIs without modifying experiment logic.

Experiment Orchestration Layer

The chapter6/android-world/experiment_core.py file implements the core experiment loop through the AndroidWorldRunner class. This engine handles task execution loops, metric aggregation, and reproducible seeding across the benchmark's 116 predefined tasks. The system supports configurable observation pipelines, injecting either raw UIAutomator hierarchies or compact representations that optimize token efficiency while maintaining task-completion fidelity.

Android Emulator Bridge

The chapter6/android-world/run_controlled_experiment.py script serves as the CLI entry point for reproducible benchmark runs. It launches agents in a Docker-based Android emulator running API 33, manages ADB connections, and translates raw UI hierarchies into structured observations that LLMs can reason about. This bridge can be toggled between observation modes to compare token efficiency against task-completion accuracy.

Key Components for Agent Evaluation

The framework organizes functionality into distinct architectural layers:

• LLM Provider Layer – Abstracts model APIs through agentbook/providers/registry.py, supporting OpenRouter and legacy implementations
• Experiment Engine – Orchestrates task loops and metric calculation via chapter6/android-world/experiment_core.py
• Android Bridge – Communicates with the emulator and produces observations through chapter6/android-world/run_controlled_experiment.py
• Validation Suite – Stores evidence JSON and markdown reports in chapter6/android-world/validation/, including reports like paired_h5c_compact_api35_20260729/report.md
• Simulation Integration – Connects to the AgentWorld layer documented in chapter8/gaia-experience/AWorld/examples/android/README.md

Running Controlled Experiments

To execute a single Android World task with the default UIAutomator observation, initialize the runner with your chosen LLM provider and task configuration:

from agentbook.providers import registry
from chapter6.android-world.experiment_core import AndroidWorldRunner

# Choose an LLM provider (e.g., OpenRouter)

llm = registry.get_provider("openrouter", model="gpt-4o-mini")

# Initialise the runner with the Android emulator image and task ID

runner = AndroidWorldRunner(
    llm=llm,
    task_id="settings_wifi_toggle",   # one of the 116 predefined tasks

    observation_mode="raw",          # can also be "compact"

    seed=42,
)

# Execute the task and obtain the result dict

result = runner.run()
print(result["success"], result["metrics"])

For comparing observation pipeline efficiency, run the same task with different modes to analyze token consumption versus success rates:

def compare_observation_modes(task_id):
    raw = AndroidWorldRunner(llm=llm, task_id=task_id,
                            observation_mode="raw").run()
    compact = AndroidWorldRunner(llm=llm, task_id=task_id,
                                observation_mode="compact").run()
    print("Raw token ratio:", raw["metrics"]["token_ratio"])
    print("Compact token ratio:", compact["metrics"]["token_ratio"])
    print("Success diff:", raw["success"] - compact["success"])

To generate comprehensive benchmark results across multiple tasks, use the AndroidWorldBenchmark class:

from chapter6.android-world.experiment_core import AndroidWorldBenchmark

benchmark = AndroidWorldBenchmark(llm=llm, seed=123)
summary = benchmark.run_all(tasks=["settings_wifi_toggle",
                                   "open_clock_app",
                                   "share_image"])
print(summary.to_markdown())

Validation Suite and Metrics

The framework tracks token ratio, latency ratio, and success rate during experiments. Validation reports stored in chapter6/android-world/validation/ document controlled smoke-run results, helping developers verify that the environment and task definitions are correct before running full benchmarks.

The requirements.txt file in the Android World directory pins exact versions of Android tooling (ADB, emulator) and Python dependencies required for deterministic runs across different environments.

Summary

• The ai-agent-book repository implements Android World evaluation through a modular architecture separating LLM providers, experiment orchestration, and Android emulation
• 116 real-world tasks test agent capabilities across settings, files, and third-party apps via the AndroidWorldRunner class in chapter6/android-world/experiment_core.py
• Dual observation modes (raw vs. compact UIAutomator) enable experiments comparing token efficiency against task-completion fidelity
• The validation suite in chapter6/android-world/validation/ provides reproducible evidence collection and environment verification
• Docker-based Android emulator integration ensures consistent API 33 environments across different development machines

Frequently Asked Questions

How do I configure the LLM provider for Android World experiments?

Initialize the provider through the registry system in agentbook/providers/registry.py. Pass the provider instance to AndroidWorldRunner along with your model selection, such as registry.get_provider("openrouter", model="gpt-4o-mini"). The abstraction layer handles authentication and request routing automatically.

What is the difference between raw and compact observation modes?

Raw observation mode transmits complete UIAutomator hierarchies with full XML node information, providing maximum context but consuming more tokens. Compact mode extracts essential interactive elements and spatial relationships, significantly reducing token ratios while maintaining sufficient state information for task completion.

Which metrics does the framework track during Android World evaluation?

The system collects token ratio (tokens consumed per task step), latency ratio (response time measurements), and success rate (binary task completion indicators). These metrics are aggregated in the result dictionary returned by runner.run() and can be exported to markdown tables via the benchmark summary interface.

How does the Android World benchmark integrate with AgentWorld?

The Android World implementation in Chapter 6 connects to the AgentWorld simulation layer documented in chapter8/gaia-experience/AWorld/. This integration allows agents to receive observations from the Android emulator via the UIAutomator bridge while maintaining compatibility with the broader GAIA (General AI Assistant) evaluation framework used throughout the repository.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}