# How to Use XcodeBuildMCP to Interact with iOS Simulator UI Elements

> Master XcodeBuildMCP to control iOS Simulator UI from the command line. Discover simulators, launch apps, and perform UI actions like taps and swipes programmatically.

- Repository: [Thomas Ricouard/Skills](https://github.com/Dimillian/Skills)
- Tags: how-to-guide
- Published: 2026-04-01

---

**Use XcodeBuildMCP commands to discover a booted simulator, configure build defaults, launch your app, and execute UI interactions like tapping, typing, and swiping via the command-line interface.**

The **Dimillian/Skills** repository provides a declarative workflow for automating iOS simulator interactions through XcodeBuildMCP (Mobile Control Protocol). According to the [`ios-debugger-agent/SKILL.md`](https://github.com/Dimillian/Skills/blob/main/ios-debugger-agent/SKILL.md) specification, you can script complete UI testing sequences—from build and launch to element interaction and log capture—without manual Xcode intervention.

## Discover the Booted Simulator

Before interacting with UI elements, you must identify a running simulator instance. The `mcp__XcodeBuildMCP__list_sims` command returns JSON data about available simulators.

```bash
mcp__XcodeBuildMCP__list_sims

```

Parse the output to select the device where `"state"` equals `"Booted"`. As specified in [`ios-debugger-agent/SKILL.md`](https://github.com/Dimillian/Skills/blob/main/ios-debugger-agent/SKILL.md) (lines 18-23), if no simulator is currently booted, you must ask the user to start one manually rather than auto-booting.

## Configure Session Defaults

Establish build parameters and target device settings using `mcp__XcodeBuildMCP__session-set-defaults`. This command stores configuration for subsequent operations.

```bash
mcp__XcodeBuildMCP__session-set-defaults \
  --projectPath /path/to/YourProject.xcodeproj \
  --scheme YourAppScheme \
  --simulatorId <booted-sim-id> \
  --configuration Debug \
  --useLatestOS true

```

**Required arguments:** `projectPath`, `scheme`, and `simulatorId` are mandatory, while `configuration` and `useLatestOS` remain optional. Reference implementation details appear in [`ios-debugger-agent/SKILL.md`](https://github.com/Dimillian/Skills/blob/main/ios-debugger-agent/SKILL.md).

## Build and Launch the Application

### Full Build and Run

Execute a complete build and deployment cycle with:

```bash
mcp__XcodeBuildMCP__build_run_sim

```

If the build fails, inspect the error output. You may retry with `--preferXcodebuild` as noted in the troubleshooting section (lines 48-52 of [`SKILL.md`](https://github.com/Dimillian/Skills/blob/main/SKILL.md)).

### Launch Without Building

For already-built apps, use:

```bash
mcp__XcodeBuildMCP__launch_app_sim

```

To obtain the bundle ID required for log capture, first retrieve the app path, then extract the identifier:

```bash
APP_PATH=$(mcp__XcodeBuildMCP__get_sim_app_path)
BUNDLE_ID=$(mcp__XcodeBuildMCP__get_app_bundle_id "$APP_PATH")

```

These helper commands appear in lines 30-33 of the agent specification.

## Verify UI Readiness

Confirm the application interface is fully loaded before attempting interactions:

```bash
mcp__XcodeBuildMCP__describe_ui   # Returns view hierarchy JSON

mcp__XcodeBuildMCP__screenshot     # Captures current screen state

```

If the hierarchy JSON is empty or the screenshot shows only the launch screen, wait briefly and retry. The `describe_ui` command is particularly critical—it provides element identifiers needed for subsequent tap and type operations.

## Interact with Simulator UI Elements

XcodeBuildMCP supports direct manipulation of interface elements through accessibility identifiers or coordinates.

### Describe the View Hierarchy

```bash
mcp__XcodeBuildMCP__describe_ui

```

This returns the complete accessibility tree, enabling you to locate `accessibilityIdentifier` and `accessibilityLabel` values for target elements.

### Tap Elements

Tap by identifier, label, or absolute coordinates:

```bash
mcp__XcodeBuildMCP__tap --id loginButton
mcp__XcodeBuildMCP__tap --label "Sign In"
mcp__XcodeBuildMCP__tap --x 150 --y 300

```

**Best practice:** Prefer `id` or `label` arguments; use coordinates only when elements lack accessibility identifiers.

### Type Text into Input Fields

First focus the field, then enter text:

```bash
mcp__XcodeBuildMCP__tap --id usernameField
mcp__XcodeBuildMCP__type_text --text "test_user" --elementId usernameField

```

The `elementId` parameter corresponds to the accessibility identifier of the target text field.

### Execute Gestures

Perform scroll or swipe actions:

```bash
mcp__XcodeBuildMCP__gesture --type scroll --direction down --distance 200
mcp__XcodeBuildMCP__gesture --type swipe --direction left --distance 300

```

Common use cases include navigating table views (`scroll`) and dismissing modal views (`swipe`).

### Capture Screenshots

Document UI state after interactions:

```bash
mcp__XcodeBuildMCP__screenshot --output ./verification.png

```

These UI interaction commands are documented in lines 34-42 of [`ios-debugger-agent/SKILL.md`](https://github.com/Dimillian/Skills/blob/main/ios-debugger-agent/SKILL.md).

## Capture Runtime Logs

Monitor application output during UI automation:

```bash

# Start log capture using previously obtained bundle ID

mcp__XcodeBuildMCP__start_sim_log_cap --bundleId "$BUNDLE_ID"

# Execute UI interactions...

mcp__XcodeBuildMCP__stop_sim_log_cap --bundleId "$BUNDLE_ID"

```

For raw console output, add `--captureConsole true` when launching the application. This workflow appears in lines 43-47 of the specification file.

## Complete Automation Example

The following bash script demonstrates an end-to-end workflow using XcodeBuildMCP to build an app, interact with login fields, and capture results:

```bash
#!/usr/bin/env bash
set -euo pipefail

# Discover booted simulator

SIM=$(mcp__XcodeBuildMCP__list_sims | jq -r '.[] | select(.state=="Booted") | .id')
[[ -z "$SIM" ]] && { echo "No booted simulator found"; exit 1; }

# Configure session

mcp__XcodeBuildMCP__session-set-defaults \
  --projectPath "$(pwd)/MyApp.xcodeproj" \
  --scheme MyApp \
  --simulatorId "$SIM"

# Build and launch

mcp__XcodeBuildMCP__build_run_sim

# Verify UI loaded

mcp__XcodeBuildMCP__describe_ui > hierarchy.json

# Interact with login form

mcp__XcodeBuildMCP__tap --id usernameField
mcp__XcodeBuildMCP__type_text --text "demo_user" --elementId usernameField
mcp__XcodeBuildMCP__tap --id passwordField
mcp__XcodeBuildMCP__type_text --text "secret123" --elementId passwordField
mcp__XcodeBuildMCP__tap --id submitButton

# Capture logs

BUNDLE=$(mcp__XcodeBuildMCP__get_app_bundle_id "$(mcp__XcodeBuildMCP__get_sim_app_path)")
mcp__XcodeBuildMCP__start_sim_log_cap --bundleId "$BUNDLE"
sleep 2
mcp__XcodeBuildMCP__stop_sim_log_cap --bundleId "$BUNDLE" > app_logs.txt

# Final verification

mcp__XcodeBuildMCP__screenshot --output final_state.png

```

This implementation follows the exact sequence specified in [`ios-debugger-agent/SKILL.md`](https://github.com/Dimillian/Skills/blob/main/ios-debugger-agent/SKILL.md) within the **Dimillian/Skills** repository.

## Troubleshooting Common Issues

- **Build failures:** Retry with `--preferXcodebuild` and prompt the user for confirmation before continuing.
- **Application not launching:** Verify the `scheme` name matches exactly in your `.xcodeproj` and confirm the bundle ID extraction succeeded.
- **Elements not found:** Re-run `describe_ui` after layout changes (such as modal presentations or orientation changes) to obtain current accessibility identifiers.

These solutions derive from the **Troubleshooting** section (lines 48-52) of the agent specification.

## Summary

- **Discovery:** Use `mcp__XcodeBuildMCP__list_sims` to identify booted simulators before any interaction.
- **Configuration:** Set mandatory defaults including `projectPath`, `scheme`, and `simulatorId` via `session-set-defaults`.
- **Execution:** Build and launch with `build_run_sim`, then verify readiness using `describe_ui` or `screenshot`.
- **Interaction:** Manipulate elements using `tap`, `type_text`, and `gesture` commands with accessibility identifiers for reliability.
- **Validation:** Capture logs with `start_sim_log_cap` and document final states with `screenshot` for debugging.

## Frequently Asked Questions

### What is XcodeBuildMCP and where is it defined?

**XcodeBuildMCP is a Mobile Control Protocol implementation for iOS automation** found in the Dimillian/Skills repository. The complete command reference and workflow specification are documented in [`ios-debugger-agent/SKILL.md`](https://github.com/Dimillian/Skills/blob/main/ios-debugger-agent/SKILL.md), while the catalog entry in [`docs/skills.json`](https://github.com/Dimillian/Skills/blob/main/docs/skills.json) surfaces the agent to the broader Skills system.

### Can I interact with UI elements using coordinates instead of identifiers?

**Yes, but accessibility identifiers are strongly preferred.** Use `--id` or `--label` arguments with the `tap` command when possible. Coordinate-based tapping (`--x` and `--y`) is supported only as a fallback for elements lacking accessibility metadata, as noted in the UI Interaction section of the specification.

### How do I capture console output while running automated UI tests?

**Start log capture before executing interactions.** Run `mcp__XcodeBuildMCP__start_sim_log_cap --bundleId <id>` prior to your UI commands, then `mcp__XcodeBuildMCP__stop_sim_log_cap` afterward to retrieve the output. Alternatively, add `--captureConsole true` during the launch phase to enable continuous console monitoring, as implemented in lines 43-47 of [`ios-debugger-agent/SKILL.md`](https://github.com/Dimillian/Skills/blob/main/ios-debugger-agent/SKILL.md).

### What should I do if the UI hierarchy returns empty JSON?

**Wait and retry.** Empty output from `describe_ui` typically indicates the application is still launching or the view controller hierarchy hasn't finished initializing. According to the workflow documentation, you should verify readiness by checking the view hierarchy or taking a screenshot, and only proceed with interactions once elements are present.