How navigator.py Interprets and Traverses the iOS Accessibility Tree to Find UI Elements

The navigator.py module interprets and traverses the iOS accessibility tree by fetching the raw hierarchical data from IDB, flattening it into a searchable list of Element objects, and applying sequential filters to locate specific UI components by type, identifier, or text content.

The ios-simulator-skill repository provides Python scripts for automating iOS Simulator interactions through Facebook's IDB (iOS Device Bridge). At the heart of this automation lies navigator.py, which must efficiently interpret and traverse the accessibility tree to locate buttons, text fields, and other interactive elements. This article examines the three-step pipeline that transforms raw accessibility data into actionable UI coordinates.

The Three-Step Pipeline for Accessibility Tree Traversal

The Navigator class in scripts/navigator.py implements a deterministic three-step process to convert the nested accessibility hierarchy into searchable UI elements.

Step 1: Fetch and Cache the Raw Tree

The pipeline begins by retrieving the complete accessibility hierarchy from the iOS Simulator. The Navigator object calls get_accessibility_tree() from common.idb_utils, which executes the IDB command:

idb ui describe-all --json --nested

This returns a deeply nested JSON structure representing the entire UI hierarchy. To optimize performance, the result is cached in self._tree_cache within the Navigator instance. Subsequent lookups reuse this cached data unless force_refresh=True is explicitly requested, minimizing expensive subprocess calls.

Step 2: Flatten the Hierarchical Structure

Raw accessibility trees are nested dictionaries where each node may contain a "children" list. Searching this structure recursively for every query would be inefficient. Instead, Navigator._flatten_tree() (lines 150-166 in navigator.py) performs a one-time recursive traversal that:

1. Walks every node in the nested structure
2. Creates an Element dataclass instance for each node containing a "type" field
3. Appends each Element to a flat Python list

This flattening operation transforms the hierarchical tree into a linear, searchable collection while preserving critical properties like frame, traits, and identifier for each UI element.

Step 3: Search and Filter Elements

With the flattened list available, Navigator.find_element() (lines 170-197) implements a sequential filtering strategy to locate specific UI components. The method iterates over the flat list and applies filters in this order:

• Enabled check: Skips disabled elements immediately
• Type matching: If element_type is specified, requires exact match against elem.type
• Identifier matching: If identifier is provided, requires exact match against elem.identifier
• Text matching: Performs fuzzy (default) or exact match against the concatenation of elem.label and elem.value

The index parameter allows selecting a specific occurrence when multiple elements match the criteria (0-based indexing).

Understanding the Element Dataclass and Interaction

The Element dataclass serves as the primary interface between the accessibility tree and automation actions. Key properties include:

• center: Computes the tap coordinate from the element's frame (calculated as the midpoint of the CGRect)
• description: Generates a human-readable string combining type, label, and identifier for logging and debugging

When automation commands like tap or text entry are requested, the navigator executes the search pipeline to obtain an Element, then issues IDB commands using the element's center coordinates:

# Internal tap implementation (lines 198-207)

idb ui tap <x> <y>

For text entry, the sequence involves first tapping the field to focus it, then sending the text via idb ui text.

Practical Code Examples

Example 1: Find and Tap a Button by Visible Text

Locate a button displaying "Login" and simulate a tap:

python scripts/navigator.py \
    --find-text "Login" \
    --tap \
    --udid <simulator-udid>

The navigator retrieves the cached tree, flattens it, matches elements whose label or value contains "Login", and taps the center of the first match.

Example 2: Find the Third Enabled Text Field and Type a Username

Target the third text field (0-based index 2) and enter text:

python scripts/navigator.py \
    --find-type TextField \
    --index 2 \
    --enter-text "alice@example.com" \
    --udid <simulator-udid>

find_element() filters by type == "TextField" and selects the element at index 2. The script first taps the field to focus it, then sends the text via idb ui text.

Example 3: List All Tappable Elements on the Current Screen

Audit available interactive elements:

python scripts/navigator.py --list --udid <simulator-udid>

Internally, navigator.list_elements() returns the full flat list. The script filters for common tappable types (Button, Link, Cell, TextField, SecureTextField) and prints each element's description and center coordinates.

Key Files and Utilities

File	Role
scripts/navigator.py	Core navigation logic – fetches, caches, flattens, searches, and interacts with UI elements.
scripts/common/idb_utils.py	Provides get_accessibility_tree and flatten_tree that power the tree handling in navigator.py.
scripts/common/__init__.py	Re-exports shared utilities used by navigator.py.

Summary

• Fetch and cache: navigator.py retrieves the accessibility tree via idb ui describe-all --json --nested and caches it to minimize subprocess overhead.
• Flatten: The _flatten_tree() method recursively transforms the nested hierarchy into a linear list of Element dataclass instances.
• Filter: find_element() applies sequential filters (enabled status, type, identifier, text) to locate specific UI components.
• Interact: Located elements expose center coordinates for tapping and descriptions for logging, enabling precise automation via IDB commands.

Frequently Asked Questions

What is the iOS accessibility tree?

The iOS accessibility tree is a hierarchical representation of the user interface exposed by the iOS Simulator through accessibility APIs. Each node represents a UI element (buttons, labels, text fields) with properties like type, label, value, frame, and traits. IDB's describe-all command exports this tree as JSON, which navigator.py consumes to locate interactive elements for automation.

Why does navigator.py flatten the tree instead of searching recursively?

Flattening converts the nested dictionary structure into a linear list during a single recursive pass. This design choice enables efficient repeated searches using simple Python list comprehensions and filters without traversing the hierarchy multiple times. According to the source code in navigator.py (lines 150-166), this approach preserves all element properties (frame, traits, identifier) while making arbitrary queries like "find the third enabled button" straightforward to implement.

How does the caching mechanism work in navigator.py?

The Navigator class maintains self._tree_cache to store the parsed accessibility tree between operations. When get_accessibility_tree() is called, it first checks this cache; only if the cache is empty or force_refresh=True is passed does it execute the expensive idb ui describe-all --json --nested subprocess call. This optimization significantly improves performance when multiple element lookups occur in sequence, as the tree is fetched once and reused for subsequent searches.

What filters can be applied when finding elements?

The find_element() method supports four sequential filters applied in order: Enabled status (skips disabled elements), Type (exact match against the element's type field), Identifier (exact match against the element's accessibility identifier), and Text (fuzzy or exact match against the concatenation of label and value). Additionally, the index parameter allows selecting a specific occurrence when multiple elements match the criteria, using 0-based indexing.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}