# Architecture Differences Between macOS AX Helper and Windows UIA Helper in pi-computer-use

> Explore macOS AX Helper vs Windows UIA Helper architecture. Understand Unix-domain sockets, Accessibility API, child processes, and UI Automation for pi-computer-use.

- Repository: [injaneity/pi-computer-use](https://github.com/injaneity/pi-computer-use)
- Tags: architecture
- Published: 2026-07-16

---

**The macOS AX helper runs as a persistent Unix-domain socket daemon using the Accessibility API with built-in cursor overlays, while the Windows UIA helper operates as a STDIN/stdout child process using Windows UI Automation and a fixed worker pool for concurrency.**

The `pi-computer-use` repository abstracts OS-specific accessibility mechanisms behind a platform-neutral backend. Two native helpers implement the low-level work: a **macOS AX helper** that interfaces with Apple's Accessibility framework, and a **Windows UIA helper** that leverages Microsoft's UI Automation API. Understanding these architecture differences is essential for debugging platform-specific behaviors and extending the system's capabilities.

## Native API Integration

### macOS Accessibility Framework

The macOS helper utilizes the **Accessibility (AX) API**, **ScreenCaptureKit**, **AppKit** for cursor rendering, and **Vision OCR** for text recognition. Native interactions occur in [`native/macos/bridge.swift`](https://github.com/injaneity/pi-computer-use/blob/main/native/macos/bridge.swift), which translates high-level commands into `AXUIElement` operations.

### Windows UI Automation

The Windows implementation relies on the **Windows UI Automation (UIA)** API alongside standard Windows capture and input mechanisms. Unlike the macOS implementation, the Windows helper does not expose a separate reference store for UI elements—it works directly with UIA element IDs.

## Transport Mechanisms and Process Architecture

### Unix-Domain Socket Daemon (macOS)

The macOS component runs as a **daemon** that listens on a **Unix-domain socket** (`bridge.sock`). Communication follows a newline-delimited JSON-lines protocol. The TypeScript client in [`src/platform/macos/helper.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/platform/macos/helper.ts) manages daemon installation and lifecycle via `ensureInstalled()` and `launchDaemon()`, implementing automatic reconnection if the daemon disappears. The protocol version is defined as `HELPER_PROTOCOL_VERSION = 6`.

### STDIN/Stdout Child Process (Windows)

Windows uses a **stand-alone executable** (`windows-bridge.exe`) spawned as a child process. Communication occurs via **STDIN/STDOUT** using the same newline-delimited JSON format. The TypeScript client in [`src/platform/windows/helper.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/platform/windows/helper.ts) checks executable presence with `isExecutable()`, starts the process on demand via `process()`, and maintains a single child process throughout the Pi session. The protocol version is `WINDOWS_HELPER_PROTOCOL_VERSION = 4`.

## Concurrency Models and Reference Storage

### Reference Store and Element Management (macOS)

The macOS daemon maintains an **AXRefStore** (implemented in Swift) that maps stable references to native `AXUIElement`s. References use prefixed strings (`w…` for windows, `e…` for elements) and include optional snapshots. The Swift implementation in [`bridge.swift`](https://github.com/injaneity/pi-computer-use/blob/main/bridge.swift) uses thread-safe storage:

```swift
// native/macos/bridge.swift
func storeElement(_ element: AXUIElement, snapshot: Snapshot? = nil) -> String {
    lock.lock()
    defer { lock.unlock() }
    nextId += 1
    let ref = "e\(nextId)"
    elements[ref] = element
    snapshots[ref] = snapshot
    return ref
}

```

Global physical-input actions are serialized inside the daemon, though the daemon can handle multiple in-flight requests correlated via an `id` field in JSON payloads.

### Worker Pool and Promise Management (Windows)

The Windows helper employs a **fixed worker pool** inside the executable. UIA-only batches do not acquire the global physical-input lock, while batches requiring pointer or keyboard input hold the lock for the entire transaction. Unlike macOS, Windows keeps a map of pending request IDs to **Promise** handlers in the TypeScript client ([`src/platform/windows/helper.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/platform/windows/helper.ts)), with the native side working directly with UIA element IDs rather than maintaining its own reference store.

## Visual Feedback and Cursor Handling

### macOS Cursor Overlay

The macOS helper implements an **overlay cursor** directly in the native layer. This visual indicator is drawn on top of captured images using AppKit in [`bridge.swift`](https://github.com/injaneity/pi-computer-use/blob/main/bridge.swift) and associated `agent_cursor` files, but never modifies the actual system cursor.

### Windows Frontend Handling

Windows provides **no built-in cursor overlay** in the helper. Visual feedback is handled entirely by the frontend in [`view.ts`](https://github.com/injaneity/pi-computer-use/blob/main/view.ts), keeping the native helper focused solely on automation and capture tasks.

## Protocol Versions and Error Handling

Both helpers report an `architectureVersion` and adhere to platform invariants verified by `assertPlatformArchitecture` in [`src/platform/architecture.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/platform/architecture.ts).

### macOS Error Handling

Errors wrap into `HelperCommandError` or `HelperTransportError`, returning JSON with `{ok:false, error:{...}}`. The client retries connection if protocol mismatches are detected against version 6.

### Windows Error Handling

Similar error objects apply, but protocol mismatch checks against `WINDOWS_HELPER_PROTOCOL_VERSION = 4`. Missing invariants cause immediate helper aborts to preserve contract parity across platforms.

## Code Examples

Initialize the macOS helper and verify architecture:

```typescript
// src/platform/macos/helper.ts
await macosHelper.ensureInstalled(signal);
const diagnostics = await macosHelper.ensureProtocol(signal);
assertPlatformArchitecture("macOS", diagnostics);

```

Spawn and command the Windows helper:

```typescript
// src/platform/windows/helper.ts
await windowsHelper.ensureInstalled(signal);
const result = await windowsHelper.command("listRoots", {}, { signal });

```

## Summary

- The **macOS helper** is a Unix-domain socket daemon using the AX API with protocol version 6, while the **Windows helper** uses a STDIN/stdout child process with UIA and protocol version 4.
- macOS maintains an **AXRefStore** for element references with prefixed IDs (`w…`, `e…`); Windows uses TypeScript-side Promise mapping with direct UIA element IDs.
- macOS provides **native cursor overlay** capabilities in [`bridge.swift`](https://github.com/injaneity/pi-computer-use/blob/main/bridge.swift); Windows delegates visual feedback to the frontend ([`view.ts`](https://github.com/injaneity/pi-computer-use/blob/main/view.ts)).
- Both helpers enforce platform invariants through `assertPlatformArchitecture` in [`src/platform/architecture.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/platform/architecture.ts), but differ in transport, serialization, and concurrency models as documented in [`docs/architecture.md`](https://github.com/injaneity/pi-computer-use/blob/main/docs/architecture.md).

## Frequently Asked Questions

### What transport mechanism does the macOS AX helper use?

The macOS AX helper communicates via a **Unix-domain socket** (`bridge.sock`) using newline-delimited JSON. It runs as a persistent daemon that the TypeScript client connects to and manages via [`src/platform/macos/helper.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/platform/macos/helper.ts).

### How does the Windows UIA helper handle concurrency differently from macOS?

The Windows helper uses a **fixed worker pool** where UIA-only batches bypass the global physical-input lock, whereas macOS serializes global physical-input actions inside the daemon. Windows maintains pending request mappings in TypeScript Promises, while macOS stores native references in the Swift AXRefStore.

### Why does the macOS helper have a cursor overlay while Windows does not?

The macOS implementation draws an overlay cursor using AppKit in [`native/macos/bridge.swift`](https://github.com/injaneity/pi-computer-use/blob/main/native/macos/bridge.swift) to visualize actions without altering the system cursor. The Windows helper lacks this feature by design; visual feedback is handled by the frontend [`view.ts`](https://github.com/injaneity/pi-computer-use/blob/main/view.ts) to keep the automation layer lightweight.

### Which files define the protocol versions for each helper?

The macOS protocol version (`HELPER_PROTOCOL_VERSION = 6`) is defined in [`src/platform/macos/helper.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/platform/macos/helper.ts), while the Windows version (`WINDOWS_HELPER_PROTOCOL_VERSION = 4`) is defined in [`src/platform/windows/helper.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/platform/windows/helper.ts).