# How navigator.py Interprets and Traverses the iOS Accessibility Tree to Find UI Elements

> Discover how navigator.py interprets the iOS accessibility tree. This module fetches, flattens, and filters data to efficiently locate UI elements by type, identifier, or text.

- Repository: [Conor/ios-simulator-skill](https://github.com/conorluddy/ios-simulator-skill)
- Tags: how-to-guide
- Published: 2026-02-27

---

**The [`navigator.py`](https://github.com/conorluddy/ios-simulator-skill/blob/main/navigator.py) module interprets and traverses the iOS accessibility tree by fetching the raw hierarchical data from IDB, flattening it into a searchable list of `Element` objects, and applying sequential filters to locate specific UI components by type, identifier, or text content.**

The `ios-simulator-skill` repository provides Python scripts for automating iOS Simulator interactions through Facebook's IDB (iOS Device Bridge). At the heart of this automation lies [`navigator.py`](https://github.com/conorluddy/ios-simulator-skill/blob/main/navigator.py), which must efficiently interpret and traverse the accessibility tree to locate buttons, text fields, and other interactive elements. This article examines the three-step pipeline that transforms raw accessibility data into actionable UI coordinates.

## The Three-Step Pipeline for Accessibility Tree Traversal

The `Navigator` class in [`scripts/navigator.py`](https://github.com/conorluddy/ios-simulator-skill/blob/main/scripts/navigator.py) implements a deterministic three-step process to convert the nested accessibility hierarchy into searchable UI elements.

### Step 1: Fetch and Cache the Raw Tree

The pipeline begins by retrieving the complete accessibility hierarchy from the iOS Simulator. The `Navigator` object calls `get_accessibility_tree()` from `common.idb_utils`, which executes the IDB command:

```bash
idb ui describe-all --json --nested

```

This returns a deeply nested JSON structure representing the entire UI hierarchy. To optimize performance, the result is cached in `self._tree_cache` within the `Navigator` instance. Subsequent lookups reuse this cached data unless `force_refresh=True` is explicitly requested, minimizing expensive subprocess calls.

### Step 2: Flatten the Hierarchical Structure

Raw accessibility trees are nested dictionaries where each node may contain a `"children"` list. Searching this structure recursively for every query would be inefficient. Instead, `Navigator._flatten_tree()` (lines 150-166 in [`navigator.py`](https://github.com/conorluddy/ios-simulator-skill/blob/main/navigator.py)) performs a one-time recursive traversal that:

1. Walks every node in the nested structure
2. Creates an `Element` dataclass instance for each node containing a `"type"` field
3. Appends each `Element` to a flat Python list

This flattening operation transforms the hierarchical tree into a linear, searchable collection while preserving critical properties like `frame`, `traits`, and `identifier` for each UI element.

### Step 3: Search and Filter Elements

With the flattened list available, `Navigator.find_element()` (lines 170-197) implements a sequential filtering strategy to locate specific UI components. The method iterates over the flat list and applies filters in this order:

- **Enabled check**: Skips disabled elements immediately
- **Type matching**: If `element_type` is specified, requires exact match against `elem.type`
- **Identifier matching**: If `identifier` is provided, requires exact match against `elem.identifier`
- **Text matching**: Performs fuzzy (default) or exact match against the concatenation of `elem.label` and `elem.value`

The `index` parameter allows selecting a specific occurrence when multiple elements match the criteria (0-based indexing).

## Understanding the Element Dataclass and Interaction

The `Element` dataclass serves as the primary interface between the accessibility tree and automation actions. Key properties include:

- **`center`**: Computes the tap coordinate from the element's frame (calculated as the midpoint of the CGRect)
- **`description`**: Generates a human-readable string combining type, label, and identifier for logging and debugging

When automation commands like tap or text entry are requested, the navigator executes the search pipeline to obtain an `Element`, then issues IDB commands using the element's center coordinates:

```python

# Internal tap implementation (lines 198-207)

idb ui tap <x> <y>

```

For text entry, the sequence involves first tapping the field to focus it, then sending the text via `idb ui text`.

## Practical Code Examples

### Example 1: Find and Tap a Button by Visible Text

Locate a button displaying "Login" and simulate a tap:

```bash
python scripts/navigator.py \
    --find-text "Login" \
    --tap \
    --udid <simulator-udid>

```

The navigator retrieves the cached tree, flattens it, matches elements whose label or value contains "Login", and taps the center of the first match.

### Example 2: Find the Third Enabled Text Field and Type a Username

Target the third text field (0-based index 2) and enter text:

```bash
python scripts/navigator.py \
    --find-type TextField \
    --index 2 \
    --enter-text "alice@example.com" \
    --udid <simulator-udid>

```

`find_element()` filters by `type == "TextField"` and selects the element at index 2. The script first taps the field to focus it, then sends the text via `idb ui text`.

### Example 3: List All Tappable Elements on the Current Screen

Audit available interactive elements:

```bash
python scripts/navigator.py --list --udid <simulator-udid>

```

Internally, `navigator.list_elements()` returns the full flat list. The script filters for common tappable types (`Button`, `Link`, `Cell`, `TextField`, `SecureTextField`) and prints each element's description and center coordinates.

## Key Files and Utilities

| File | Role |
|------|------|
| [`scripts/navigator.py`](https://github.com/conorluddy/ios-simulator-skill/blob/main/scripts/navigator.py) | Core navigation logic – fetches, caches, flattens, searches, and interacts with UI elements. |
| [`scripts/common/idb_utils.py`](https://github.com/conorluddy/ios-simulator-skill/blob/main/scripts/common/idb_utils.py) | Provides `get_accessibility_tree` and `flatten_tree` that power the tree handling in [`navigator.py`](https://github.com/conorluddy/ios-simulator-skill/blob/main/navigator.py). |
| [`scripts/common/__init__.py`](https://github.com/conorluddy/ios-simulator-skill/blob/main/scripts/common/__init__.py) | Re-exports shared utilities used by [`navigator.py`](https://github.com/conorluddy/ios-simulator-skill/blob/main/navigator.py). |

## Summary

- **Fetch and cache**: [`navigator.py`](https://github.com/conorluddy/ios-simulator-skill/blob/main/navigator.py) retrieves the accessibility tree via `idb ui describe-all --json --nested` and caches it to minimize subprocess overhead.
- **Flatten**: The `_flatten_tree()` method recursively transforms the nested hierarchy into a linear list of `Element` dataclass instances.
- **Filter**: `find_element()` applies sequential filters (enabled status, type, identifier, text) to locate specific UI components.
- **Interact**: Located elements expose `center` coordinates for tapping and descriptions for logging, enabling precise automation via IDB commands.

## Frequently Asked Questions

### What is the iOS accessibility tree?

The iOS accessibility tree is a hierarchical representation of the user interface exposed by the iOS Simulator through accessibility APIs. Each node represents a UI element (buttons, labels, text fields) with properties like `type`, `label`, `value`, `frame`, and `traits`. IDB's `describe-all` command exports this tree as JSON, which [`navigator.py`](https://github.com/conorluddy/ios-simulator-skill/blob/main/navigator.py) consumes to locate interactive elements for automation.

### Why does navigator.py flatten the tree instead of searching recursively?

Flattening converts the nested dictionary structure into a linear list during a single recursive pass. This design choice enables efficient repeated searches using simple Python list comprehensions and filters without traversing the hierarchy multiple times. According to the source code in [`navigator.py`](https://github.com/conorluddy/ios-simulator-skill/blob/main/navigator.py) (lines 150-166), this approach preserves all element properties (`frame`, `traits`, `identifier`) while making arbitrary queries like "find the third enabled button" straightforward to implement.

### How does the caching mechanism work in navigator.py?

The `Navigator` class maintains `self._tree_cache` to store the parsed accessibility tree between operations. When `get_accessibility_tree()` is called, it first checks this cache; only if the cache is empty or `force_refresh=True` is passed does it execute the expensive `idb ui describe-all --json --nested` subprocess call. This optimization significantly improves performance when multiple element lookups occur in sequence, as the tree is fetched once and reused for subsequent searches.

### What filters can be applied when finding elements?

The `find_element()` method supports four sequential filters applied in order: **Enabled status** (skips disabled elements), **Type** (exact match against the element's `type` field), **Identifier** (exact match against the element's accessibility identifier), and **Text** (fuzzy or exact match against the concatenation of `label` and `value`). Additionally, the `index` parameter allows selecting a specific occurrence when multiple elements match the criteria, using 0-based indexing.