How to Use XcodeBuildMCP to Interact with iOS Simulator UI Elements

Use XcodeBuildMCP commands to discover a booted simulator, configure build defaults, launch your app, and execute UI interactions like tapping, typing, and swiping via the command-line interface.

The Dimillian/Skills repository provides a declarative workflow for automating iOS simulator interactions through XcodeBuildMCP (Mobile Control Protocol). According to the ios-debugger-agent/SKILL.md specification, you can script complete UI testing sequences—from build and launch to element interaction and log capture—without manual Xcode intervention.

Discover the Booted Simulator

Before interacting with UI elements, you must identify a running simulator instance. The mcp__XcodeBuildMCP__list_sims command returns JSON data about available simulators.

mcp__XcodeBuildMCP__list_sims

Parse the output to select the device where "state" equals "Booted". As specified in ios-debugger-agent/SKILL.md (lines 18-23), if no simulator is currently booted, you must ask the user to start one manually rather than auto-booting.

Configure Session Defaults

Establish build parameters and target device settings using mcp__XcodeBuildMCP__session-set-defaults. This command stores configuration for subsequent operations.

mcp__XcodeBuildMCP__session-set-defaults \
  --projectPath /path/to/YourProject.xcodeproj \
  --scheme YourAppScheme \
  --simulatorId <booted-sim-id> \
  --configuration Debug \
  --useLatestOS true

Required arguments: projectPath, scheme, and simulatorId are mandatory, while configuration and useLatestOS remain optional. Reference implementation details appear in ios-debugger-agent/SKILL.md.

Build and Launch the Application

Full Build and Run

Execute a complete build and deployment cycle with:

mcp__XcodeBuildMCP__build_run_sim

If the build fails, inspect the error output. You may retry with --preferXcodebuild as noted in the troubleshooting section (lines 48-52 of SKILL.md).

Launch Without Building

For already-built apps, use:

mcp__XcodeBuildMCP__launch_app_sim

To obtain the bundle ID required for log capture, first retrieve the app path, then extract the identifier:

APP_PATH=$(mcp__XcodeBuildMCP__get_sim_app_path)
BUNDLE_ID=$(mcp__XcodeBuildMCP__get_app_bundle_id "$APP_PATH")

These helper commands appear in lines 30-33 of the agent specification.

Verify UI Readiness

Confirm the application interface is fully loaded before attempting interactions:

mcp__XcodeBuildMCP__describe_ui   # Returns view hierarchy JSON

mcp__XcodeBuildMCP__screenshot     # Captures current screen state

If the hierarchy JSON is empty or the screenshot shows only the launch screen, wait briefly and retry. The describe_ui command is particularly critical—it provides element identifiers needed for subsequent tap and type operations.

Interact with Simulator UI Elements

XcodeBuildMCP supports direct manipulation of interface elements through accessibility identifiers or coordinates.

Describe the View Hierarchy

mcp__XcodeBuildMCP__describe_ui

This returns the complete accessibility tree, enabling you to locate accessibilityIdentifier and accessibilityLabel values for target elements.

Tap Elements

Tap by identifier, label, or absolute coordinates:

mcp__XcodeBuildMCP__tap --id loginButton
mcp__XcodeBuildMCP__tap --label "Sign In"
mcp__XcodeBuildMCP__tap --x 150 --y 300

Best practice: Prefer id or label arguments; use coordinates only when elements lack accessibility identifiers.

Type Text into Input Fields

First focus the field, then enter text:

mcp__XcodeBuildMCP__tap --id usernameField
mcp__XcodeBuildMCP__type_text --text "test_user" --elementId usernameField

The elementId parameter corresponds to the accessibility identifier of the target text field.

Execute Gestures

Perform scroll or swipe actions:

mcp__XcodeBuildMCP__gesture --type scroll --direction down --distance 200
mcp__XcodeBuildMCP__gesture --type swipe --direction left --distance 300

Common use cases include navigating table views (scroll) and dismissing modal views (swipe).

Capture Screenshots

Document UI state after interactions:

mcp__XcodeBuildMCP__screenshot --output ./verification.png

These UI interaction commands are documented in lines 34-42 of ios-debugger-agent/SKILL.md.

Capture Runtime Logs

Monitor application output during UI automation:

# Start log capture using previously obtained bundle ID

mcp__XcodeBuildMCP__start_sim_log_cap --bundleId "$BUNDLE_ID"

# Execute UI interactions...

mcp__XcodeBuildMCP__stop_sim_log_cap --bundleId "$BUNDLE_ID"

For raw console output, add --captureConsole true when launching the application. This workflow appears in lines 43-47 of the specification file.

Complete Automation Example

The following bash script demonstrates an end-to-end workflow using XcodeBuildMCP to build an app, interact with login fields, and capture results:

#!/usr/bin/env bash
set -euo pipefail

# Discover booted simulator

SIM=$(mcp__XcodeBuildMCP__list_sims | jq -r '.[] | select(.state=="Booted") | .id')
[[ -z "$SIM" ]] && { echo "No booted simulator found"; exit 1; }

# Configure session

mcp__XcodeBuildMCP__session-set-defaults \
  --projectPath "$(pwd)/MyApp.xcodeproj" \
  --scheme MyApp \
  --simulatorId "$SIM"

# Build and launch

mcp__XcodeBuildMCP__build_run_sim

# Verify UI loaded

mcp__XcodeBuildMCP__describe_ui > hierarchy.json

# Interact with login form

mcp__XcodeBuildMCP__tap --id usernameField
mcp__XcodeBuildMCP__type_text --text "demo_user" --elementId usernameField
mcp__XcodeBuildMCP__tap --id passwordField
mcp__XcodeBuildMCP__type_text --text "secret123" --elementId passwordField
mcp__XcodeBuildMCP__tap --id submitButton

# Capture logs

BUNDLE=$(mcp__XcodeBuildMCP__get_app_bundle_id "$(mcp__XcodeBuildMCP__get_sim_app_path)")
mcp__XcodeBuildMCP__start_sim_log_cap --bundleId "$BUNDLE"
sleep 2
mcp__XcodeBuildMCP__stop_sim_log_cap --bundleId "$BUNDLE" > app_logs.txt

# Final verification

mcp__XcodeBuildMCP__screenshot --output final_state.png

This implementation follows the exact sequence specified in ios-debugger-agent/SKILL.md within the Dimillian/Skills repository.

Troubleshooting Common Issues

• Build failures: Retry with --preferXcodebuild and prompt the user for confirmation before continuing.
• Application not launching: Verify the scheme name matches exactly in your .xcodeproj and confirm the bundle ID extraction succeeded.
• Elements not found: Re-run describe_ui after layout changes (such as modal presentations or orientation changes) to obtain current accessibility identifiers.

These solutions derive from the Troubleshooting section (lines 48-52) of the agent specification.

Summary

• Discovery: Use mcp__XcodeBuildMCP__list_sims to identify booted simulators before any interaction.
• Configuration: Set mandatory defaults including projectPath, scheme, and simulatorId via session-set-defaults.
• Execution: Build and launch with build_run_sim, then verify readiness using describe_ui or screenshot.
• Interaction: Manipulate elements using tap, type_text, and gesture commands with accessibility identifiers for reliability.
• Validation: Capture logs with start_sim_log_cap and document final states with screenshot for debugging.

Frequently Asked Questions

What is XcodeBuildMCP and where is it defined?

XcodeBuildMCP is a Mobile Control Protocol implementation for iOS automation found in the Dimillian/Skills repository. The complete command reference and workflow specification are documented in ios-debugger-agent/SKILL.md, while the catalog entry in docs/skills.json surfaces the agent to the broader Skills system.

Can I interact with UI elements using coordinates instead of identifiers?

Yes, but accessibility identifiers are strongly preferred. Use --id or --label arguments with the tap command when possible. Coordinate-based tapping (--x and --y) is supported only as a fallback for elements lacking accessibility metadata, as noted in the UI Interaction section of the specification.

How do I capture console output while running automated UI tests?

Start log capture before executing interactions. Run mcp__XcodeBuildMCP__start_sim_log_cap --bundleId <id> prior to your UI commands, then mcp__XcodeBuildMCP__stop_sim_log_cap afterward to retrieve the output. Alternatively, add --captureConsole true during the launch phase to enable continuous console monitoring, as implemented in lines 43-47 of ios-debugger-agent/SKILL.md.

What should I do if the UI hierarchy returns empty JSON?

Wait and retry. Empty output from describe_ui typically indicates the application is still launching or the view controller hierarchy hasn't finished initializing. According to the workflow documentation, you should verify readiness by checking the view hierarchy or taking a screenshot, and only proceed with interactions once elements are present.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}