Architecture Differences Between macOS AX Helper and Windows UIA Helper in pi-computer-use

The macOS AX helper runs as a persistent Unix-domain socket daemon using the Accessibility API with built-in cursor overlays, while the Windows UIA helper operates as a STDIN/stdout child process using Windows UI Automation and a fixed worker pool for concurrency.

The pi-computer-use repository abstracts OS-specific accessibility mechanisms behind a platform-neutral backend. Two native helpers implement the low-level work: a macOS AX helper that interfaces with Apple's Accessibility framework, and a Windows UIA helper that leverages Microsoft's UI Automation API. Understanding these architecture differences is essential for debugging platform-specific behaviors and extending the system's capabilities.

Native API Integration

macOS Accessibility Framework

The macOS helper utilizes the Accessibility (AX) API, ScreenCaptureKit, AppKit for cursor rendering, and Vision OCR for text recognition. Native interactions occur in native/macos/bridge.swift, which translates high-level commands into AXUIElement operations.

Windows UI Automation

The Windows implementation relies on the Windows UI Automation (UIA) API alongside standard Windows capture and input mechanisms. Unlike the macOS implementation, the Windows helper does not expose a separate reference store for UI elements—it works directly with UIA element IDs.

Transport Mechanisms and Process Architecture

Unix-Domain Socket Daemon (macOS)

The macOS component runs as a daemon that listens on a Unix-domain socket (bridge.sock). Communication follows a newline-delimited JSON-lines protocol. The TypeScript client in src/platform/macos/helper.ts manages daemon installation and lifecycle via ensureInstalled() and launchDaemon(), implementing automatic reconnection if the daemon disappears. The protocol version is defined as HELPER_PROTOCOL_VERSION = 6.

STDIN/Stdout Child Process (Windows)

Windows uses a stand-alone executable (windows-bridge.exe) spawned as a child process. Communication occurs via STDIN/STDOUT using the same newline-delimited JSON format. The TypeScript client in src/platform/windows/helper.ts checks executable presence with isExecutable(), starts the process on demand via process(), and maintains a single child process throughout the Pi session. The protocol version is WINDOWS_HELPER_PROTOCOL_VERSION = 4.

Concurrency Models and Reference Storage

Reference Store and Element Management (macOS)

The macOS daemon maintains an AXRefStore (implemented in Swift) that maps stable references to native AXUIElements. References use prefixed strings (w… for windows, e… for elements) and include optional snapshots. The Swift implementation in bridge.swift uses thread-safe storage:

// native/macos/bridge.swift
func storeElement(_ element: AXUIElement, snapshot: Snapshot? = nil) -> String {
    lock.lock()
    defer { lock.unlock() }
    nextId += 1
    let ref = "e\(nextId)"
    elements[ref] = element
    snapshots[ref] = snapshot
    return ref
}

Global physical-input actions are serialized inside the daemon, though the daemon can handle multiple in-flight requests correlated via an id field in JSON payloads.

Worker Pool and Promise Management (Windows)

The Windows helper employs a fixed worker pool inside the executable. UIA-only batches do not acquire the global physical-input lock, while batches requiring pointer or keyboard input hold the lock for the entire transaction. Unlike macOS, Windows keeps a map of pending request IDs to Promise handlers in the TypeScript client (src/platform/windows/helper.ts), with the native side working directly with UIA element IDs rather than maintaining its own reference store.

Visual Feedback and Cursor Handling

macOS Cursor Overlay

The macOS helper implements an overlay cursor directly in the native layer. This visual indicator is drawn on top of captured images using AppKit in bridge.swift and associated agent_cursor files, but never modifies the actual system cursor.

Windows Frontend Handling

Windows provides no built-in cursor overlay in the helper. Visual feedback is handled entirely by the frontend in view.ts, keeping the native helper focused solely on automation and capture tasks.

Protocol Versions and Error Handling

Both helpers report an architectureVersion and adhere to platform invariants verified by assertPlatformArchitecture in src/platform/architecture.ts.

macOS Error Handling

Errors wrap into HelperCommandError or HelperTransportError, returning JSON with {ok:false, error:{...}}. The client retries connection if protocol mismatches are detected against version 6.

Windows Error Handling

Similar error objects apply, but protocol mismatch checks against WINDOWS_HELPER_PROTOCOL_VERSION = 4. Missing invariants cause immediate helper aborts to preserve contract parity across platforms.

Code Examples

Initialize the macOS helper and verify architecture:

// src/platform/macos/helper.ts
await macosHelper.ensureInstalled(signal);
const diagnostics = await macosHelper.ensureProtocol(signal);
assertPlatformArchitecture("macOS", diagnostics);

Spawn and command the Windows helper:

// src/platform/windows/helper.ts
await windowsHelper.ensureInstalled(signal);
const result = await windowsHelper.command("listRoots", {}, { signal });

Summary

• The macOS helper is a Unix-domain socket daemon using the AX API with protocol version 6, while the Windows helper uses a STDIN/stdout child process with UIA and protocol version 4.
• macOS maintains an AXRefStore for element references with prefixed IDs (w…, e…); Windows uses TypeScript-side Promise mapping with direct UIA element IDs.
• macOS provides native cursor overlay capabilities in bridge.swift; Windows delegates visual feedback to the frontend (view.ts).
• Both helpers enforce platform invariants through assertPlatformArchitecture in src/platform/architecture.ts, but differ in transport, serialization, and concurrency models as documented in docs/architecture.md.

Frequently Asked Questions

What transport mechanism does the macOS AX helper use?

The macOS AX helper communicates via a Unix-domain socket (bridge.sock) using newline-delimited JSON. It runs as a persistent daemon that the TypeScript client connects to and manages via src/platform/macos/helper.ts.

How does the Windows UIA helper handle concurrency differently from macOS?

The Windows helper uses a fixed worker pool where UIA-only batches bypass the global physical-input lock, whereas macOS serializes global physical-input actions inside the daemon. Windows maintains pending request mappings in TypeScript Promises, while macOS stores native references in the Swift AXRefStore.

Why does the macOS helper have a cursor overlay while Windows does not?

The macOS implementation draws an overlay cursor using AppKit in native/macos/bridge.swift to visualize actions without altering the system cursor. The Windows helper lacks this feature by design; visual feedback is handled by the frontend view.ts to keep the automation layer lightweight.

Which files define the protocol versions for each helper?

The macOS protocol version (HELPER_PROTOCOL_VERSION = 6) is defined in src/platform/macos/helper.ts, while the Windows version (WINDOWS_HELPER_PROTOCOL_VERSION = 4) is defined in src/platform/windows/helper.ts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}