How the Element Dataclass Captures Spatial Information in iOS Simulator Automation

The Element dataclass stores spatial data in a frame dictionary containing x, y, width, and height coordinates, and exposes a center property that calculates the midpoint for tapping interactions.

The Element dataclass defined in conorluddy/ios-simulator-skill models UI elements extracted from the iOS accessibility tree, enabling precise geometric interactions with simulator interfaces. By encapsulating frame data and coordinate calculation logic, this component bridges the gap between accessibility node metadata and actionable screen coordinates for automation scripts.

The Frame Attribute: Core Spatial Storage

Spatial information in the Element dataclass resides in the frame attribute, a dict[str, float] that captures the element's geometric footprint on the device screen. The dataclass definition in ios-simulator-skill/scripts/navigator.py (lines 73-84) declares this structure explicitly:

@dataclass
class Element:
    type: str
    label: str | None
    value: str | None
    identifier: str | None
    frame: dict[str, float]      # ← holds x, y, width, height

    traits: list[str]
    enabled: bool = True

When the Navigator class flattens the accessibility tree, it constructs Element instances by extracting the raw frame dictionary directly from each node's frame field:

element = Element(
    type=node.get("type", "Unknown"),
    label=node.get("AXLabel"),
    value=node.get("AXValue"),
    identifier=node.get("AXUniqueId"),
    frame=node.get("frame", {}),          # ← populated from the tree

    traits=node.get("traits", []),
    enabled=node.get("enabled", True),
)

The frame dictionary contains four required floating-point values measured in points: x (horizontal origin), y (vertical origin), width (horizontal extent), and height (vertical extent).

Converting Frames to Actionable Coordinates

To translate static frame data into interactive tap points, the Element dataclass provides a center property that computes the geometric midpoint of the rectangle. This property returns a tuple[int, int] representing integer screen coordinates suitable for idb ui tap commands:

@property
def center(self) -> tuple[int, int]:
    """Calculate center point for tapping."""
    x = int(self.frame["x"] + self.frame["width"] / 2)
    y = int(self.frame["y"] + self.frame["height"] / 2)
    return (x, y)

The calculation performs half-width and half-height additions to the origin coordinates, then converts the results to integers to match the discrete pixel grid expected by the iOS simulator's input system.

Practical Usage Examples

Creating Elements Manually

You can instantiate Element objects directly when simulating accessibility data or building test fixtures:

from ios_simulator_skill.scripts.navigator import Element

# Simulated frame returned by the accessibility tree

frame_info = {"x": 120.0, "y": 250.0, "width": 80.0, "height": 30.0}

button = Element(
    type="Button",
    label="Submit",
    value=None,
    identifier="submitBtn",
    frame=frame_info,
    traits=["clickable"],
)

print(button.center)          # → (160, 265)

print(button.description)     # → Button "Submit"

Finding and Tapping Elements via Navigator

In production automation, the Navigator class handles element discovery and interaction:

from ios_simulator_skill.scripts.navigator import Navigator

nav = Navigator(udid=None)               # auto-detects the booted simulator

element = nav.find_element(text="Login")  # returns an Element instance

if element:
    print(f"Found {element.description} at {element.frame}")
    print(f"Tap coordinates: {element.center}")
    nav.tap(element)                     # uses the calculated center

Custom Coordinate Transformations

For scenarios requiring offset taps or coordinate adjustments, access the raw frame data to compute modified positions:

# Offset the tap by 5 points horizontally from center

x, y = element.center
adjusted = (x + 5, y)

nav.tap_at(*adjusted)    # direct tap at the shifted position

Summary

• The frame attribute in the Element dataclass stores spatial data as a dictionary with x, y, width, and height keys measured in points.
• The center property calculates the geometric midpoint by adding half-dimensions to origin coordinates and returning integer tuples.
• The Navigator class populates frame data directly from iOS accessibility tree nodes when constructing Element instances.
• All spatial calculations occur in ios-simulator-skill/scripts/navigator.py, which serves as the primary interface for coordinate-based simulator interactions.

Frequently Asked Questions

What format does the frame dictionary use?

The frame dictionary uses string keys mapped to floating-point values: {"x": float, "y": float, "width": float, "height": float}. All measurements represent points on the device screen coordinate system, with the origin (0,0) located at the top-left corner.

How does the center property calculate coordinates?

The center property computes the horizontal midpoint by adding half the width to the x-coordinate, and the vertical midpoint by adding half the height to the y-coordinate. It then converts both results to integers using Python's int() constructor to ensure compatibility with discrete pixel-based input systems.

Can I access the raw frame data directly?

Yes, the frame attribute is a public dictionary accessible as element.frame. You can read individual values directly (e.g., element.frame["x"]) or use the entire dictionary for custom geometric calculations beyond the standard center property.

Where is the Element dataclass defined?

The Element dataclass is defined in ios-simulator-skill/scripts/navigator.py at lines 73-84 according to the source code. This file also contains the Navigator class responsible for building Element instances from accessibility tree nodes and executing tap commands based on the calculated coordinates.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}