# Evaluating Agents on Mobile App Automation with Android World: A Complete Framework

> Evaluate AI agents on mobile app automation with Android World. Explore a modular framework for benchmarking autonomous agents on real-world tasks with controlled experiments and reproducible metrics.

- Repository: [Bojie Li/ai-agent-book](https://github.com/bojieli/ai-agent-book)
- Tags: tutorial
- Published: 2026-07-31

---

**The bojieli/ai-agent-book repository provides a modular framework for benchmarking autonomous AI agents on 116 real-world Android tasks using controlled experiments with configurable observation pipelines and reproducible metrics.**

Evaluating agents on mobile app automation with Android World requires a standardized environment that can simulate realistic device interactions while measuring LLM performance accurately. The ai-agent-book repository implements exactly this through its Chapter 6 Android World benchmark, which integrates tightly with the AgentWorld simulation layer to provide deterministic testing across settings management, file operations, and third-party app interactions.

## Android World Benchmark Architecture

The Android World implementation in `chapter6/android-world/` provides a comprehensive suite for testing agent capabilities on actual Android devices through a Docker-based emulator environment.

### Core Library and Provider Abstraction

The `agentbook/` directory contains the unified provider interface that abstracts LLM backends through a central registry pattern. Key files include [`agentbook/providers/registry.py`](https://github.com/bojieli/ai-agent-book/blob/main/agentbook/providers/registry.py), which maps provider names to concrete implementations, along with [`openrouter.py`](https://github.com/bojieli/ai-agent-book/blob/main/openrouter.py), [`models.py`](https://github.com/bojieli/ai-agent-book/blob/main/models.py), [`legacy.py`](https://github.com/bojieli/ai-agent-book/blob/main/legacy.py), and [`resolution.py`](https://github.com/bojieli/ai-agent-book/blob/main/resolution.py) for handling request routing and model selection. This abstraction allows seamless switching between OpenRouter and legacy model APIs without modifying experiment logic.

### Experiment Orchestration Layer

The [`chapter6/android-world/experiment_core.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter6/android-world/experiment_core.py) file implements the core experiment loop through the `AndroidWorldRunner` class. This engine handles task execution loops, metric aggregation, and reproducible seeding across the benchmark's 116 predefined tasks. The system supports configurable observation pipelines, injecting either raw UIAutomator hierarchies or compact representations that optimize token efficiency while maintaining task-completion fidelity.

### Android Emulator Bridge

The [`chapter6/android-world/run_controlled_experiment.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter6/android-world/run_controlled_experiment.py) script serves as the CLI entry point for reproducible benchmark runs. It launches agents in a Docker-based Android emulator running API 33, manages ADB connections, and translates raw UI hierarchies into structured observations that LLMs can reason about. This bridge can be toggled between observation modes to compare token efficiency against task-completion accuracy.

## Key Components for Agent Evaluation

The framework organizes functionality into distinct architectural layers:

- **LLM Provider Layer** – Abstracts model APIs through [`agentbook/providers/registry.py`](https://github.com/bojieli/ai-agent-book/blob/main/agentbook/providers/registry.py), supporting OpenRouter and legacy implementations
- **Experiment Engine** – Orchestrates task loops and metric calculation via [`chapter6/android-world/experiment_core.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter6/android-world/experiment_core.py)
- **Android Bridge** – Communicates with the emulator and produces observations through [`chapter6/android-world/run_controlled_experiment.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter6/android-world/run_controlled_experiment.py)
- **Validation Suite** – Stores evidence JSON and markdown reports in `chapter6/android-world/validation/`, including reports like [`paired_h5c_compact_api35_20260729/report.md`](https://github.com/bojieli/ai-agent-book/blob/main/paired_h5c_compact_api35_20260729/report.md)
- **Simulation Integration** – Connects to the AgentWorld layer documented in [`chapter8/gaia-experience/AWorld/examples/android/README.md`](https://github.com/bojieli/ai-agent-book/blob/main/chapter8/gaia-experience/AWorld/examples/android/README.md)

## Running Controlled Experiments

To execute a single Android World task with the default UIAutomator observation, initialize the runner with your chosen LLM provider and task configuration:

```python
from agentbook.providers import registry
from chapter6.android-world.experiment_core import AndroidWorldRunner

# Choose an LLM provider (e.g., OpenRouter)

llm = registry.get_provider("openrouter", model="gpt-4o-mini")

# Initialise the runner with the Android emulator image and task ID

runner = AndroidWorldRunner(
    llm=llm,
    task_id="settings_wifi_toggle",   # one of the 116 predefined tasks

    observation_mode="raw",          # can also be "compact"

    seed=42,
)

# Execute the task and obtain the result dict

result = runner.run()
print(result["success"], result["metrics"])

```

For comparing observation pipeline efficiency, run the same task with different modes to analyze token consumption versus success rates:

```python
def compare_observation_modes(task_id):
    raw = AndroidWorldRunner(llm=llm, task_id=task_id,
                            observation_mode="raw").run()
    compact = AndroidWorldRunner(llm=llm, task_id=task_id,
                                observation_mode="compact").run()
    print("Raw token ratio:", raw["metrics"]["token_ratio"])
    print("Compact token ratio:", compact["metrics"]["token_ratio"])
    print("Success diff:", raw["success"] - compact["success"])

```

To generate comprehensive benchmark results across multiple tasks, use the `AndroidWorldBenchmark` class:

```python
from chapter6.android-world.experiment_core import AndroidWorldBenchmark

benchmark = AndroidWorldBenchmark(llm=llm, seed=123)
summary = benchmark.run_all(tasks=["settings_wifi_toggle",
                                   "open_clock_app",
                                   "share_image"])
print(summary.to_markdown())

```

## Validation Suite and Metrics

The framework tracks **token ratio**, **latency ratio**, and **success rate** during experiments. Validation reports stored in `chapter6/android-world/validation/` document controlled smoke-run results, helping developers verify that the environment and task definitions are correct before running full benchmarks.

The [`requirements.txt`](https://github.com/bojieli/ai-agent-book/blob/main/requirements.txt) file in the Android World directory pins exact versions of Android tooling (ADB, emulator) and Python dependencies required for deterministic runs across different environments.

## Summary

- The ai-agent-book repository implements Android World evaluation through a modular architecture separating LLM providers, experiment orchestration, and Android emulation
- **116 real-world tasks** test agent capabilities across settings, files, and third-party apps via the `AndroidWorldRunner` class in [`chapter6/android-world/experiment_core.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter6/android-world/experiment_core.py)
- **Dual observation modes** (raw vs. compact UIAutomator) enable experiments comparing token efficiency against task-completion fidelity
- The validation suite in `chapter6/android-world/validation/` provides reproducible evidence collection and environment verification
- Docker-based Android emulator integration ensures consistent API 33 environments across different development machines

## Frequently Asked Questions

### How do I configure the LLM provider for Android World experiments?

Initialize the provider through the registry system in [`agentbook/providers/registry.py`](https://github.com/bojieli/ai-agent-book/blob/main/agentbook/providers/registry.py). Pass the provider instance to `AndroidWorldRunner` along with your model selection, such as `registry.get_provider("openrouter", model="gpt-4o-mini")`. The abstraction layer handles authentication and request routing automatically.

### What is the difference between raw and compact observation modes?

**Raw** observation mode transmits complete UIAutomator hierarchies with full XML node information, providing maximum context but consuming more tokens. **Compact** mode extracts essential interactive elements and spatial relationships, significantly reducing token ratios while maintaining sufficient state information for task completion.

### Which metrics does the framework track during Android World evaluation?

The system collects **token ratio** (tokens consumed per task step), **latency ratio** (response time measurements), and **success rate** (binary task completion indicators). These metrics are aggregated in the result dictionary returned by `runner.run()` and can be exported to markdown tables via the benchmark summary interface.

### How does the Android World benchmark integrate with AgentWorld?

The Android World implementation in Chapter 6 connects to the **AgentWorld** simulation layer documented in `chapter8/gaia-experience/AWorld/`. This integration allows agents to receive observations from the Android emulator via the UIAutomator bridge while maintaining compatibility with the broader GAIA (General AI Assistant) evaluation framework used throughout the repository.