How MCP (Model Context Protocol) Integration Works in OpenWork: Architecture and Implementation

OpenWork implements MCP integration through a Swift-based JSON-RPC server (MCPServer.swift) that bridges LLM agents with macOS UI automation via stdin/stdout communication.

The OpenWork project leverages the Model Context Protocol (MCP) to enable LLM-driven agents to interact with the macOS operating system through a lightweight, structured interface. At the heart of this MCP integration lies the HandsFree binary, an actor-based server that translates JSON-RPC requests into native UI actions. This architecture allows AI agents to capture screenshots, click UI elements, and inject keyboard input without requiring direct OS bindings.

MCP Server Architecture and Lifecycle

The core of OpenWork's MCP integration resides in packages/handsfree/native/HandsFree/Sources/ComputerUse/MCPServer.swift. This file defines an actor MCPServer that manages the entire request-response lifecycle through standard input and output streams.

The server initializes with two primary dependencies:

actor MCPServer {
    private let runtime = ComputerUseRuntime()
    private let input = InputService()
    // ...
}

The run() method establishes a continuous loop that reads JSON-RPC messages from stdin:

func run() async {
    log("Computer Use server starting")
    while let line = readLine(strippingNewline: true) {
        // Parse JSON and extract fields
        let id = json["id"]
        let method = json["method"] as? String ?? ""
        let params = json["params"] as? [String: Any] ?? [:]
        switch method { 
            // Dispatch to appropriate handler
        }
    }
}

Empty or malformed lines are ignored with logging, while valid requests are dispatched based on the method field. The server echoes the id field in all responses to maintain request correlation.

JSON-RPC Method Handling

The server implements four primary JSON-RPC methods as defined in lines 25-55 of MCPServer.swift:

• initialize (lines 25-32): Returns protocol version, server capabilities, and metadata to establish the connection.
• tools/list (lines 34-55): Supplies an array of supported tool schemas generated by toolSchemas().
• tools/call (lines 36-41): Executes a specific tool by name with provided arguments, returning structured results.
• notifications/initialized (lines 32-34): Currently a no-op reserved for future client-side notification handling.

When an unknown method arrives and the request contains an id, the server returns a standard JSON-RPC error with code -32601 (Method not found) as implemented in lines 42-45.

Tool Schema Definition and Registration

Each available tool is defined using the toolSchema(name:description:properties:) helper function. These schemas conform to JSON Schema specifications and include:

• name: The RPC method identifier used in tools/call invocations.
• description: Human-readable documentation for client UI rendering.
• inputSchema: A typed object definition specifying required and optional parameters.

All schemas are aggregated in toolSchemas() and returned by the tools/list method (lines 49-55). The snapshot tool schema exemplifies this structure:

toolSchema(
    name: "snapshot",
    description: "Return target-window screenshot plus compact semantic AX state …",
    properties: [
        "app": ["type": "string", "description": "Optional running app name …"],
        "pid": ["type": "number", "description": "Optional process id …"],
        // ...
    ]
)

Runtime Execution and UI Automation

The executeTool(name:args:) function serves as the dispatcher that maps incoming tool names to concrete macOS automation calls (lines 61-83). This method uses a switch statement to route commands to either ComputerUseRuntime or InputService.

ComputerUseRuntime encapsulates high-level UI actions including snapshot, click, type, and pressKey. It interfaces directly with macOS Accessibility APIs, CoreGraphics events, and ScreenCaptureKit.

InputService provides low-level mouse and keyboard event generation for CUA-compatibility tools.

The execution flow follows this pattern:

switch name {
case "snapshot":
    let result = try await runtime.snapshot(args: args)
    return jsonResult(id: id, result: result)
case "click":
    let result = try await runtime.click(
        x: intArg(args, "x"),
        y: intArg(args, "y"),
        strict: boolArg(args, "strict")
    )
    return jsonResult(id: id, result: result)
// ...
}

All execution branches are wrapped in do/catch blocks. Thrown errors are converted to structured payloads via errorPayload(_:) (lines 83-86), enriching responses with context such as missing permissions or strict mode violations.

Security, Permissions, and Strict Mode

OpenWork's MCP integration includes a strict background mode that prevents UI actions from activating foreground applications. This safety feature is exposed through boolean parameters in tool schemas and enforced in runtime calls.

When strict mode is enabled via arguments like "strict": true, the server rejects actions that would require foreground activation, returning errors containing "strictMode" metadata (lines 77-80). This ensures agents can operate without disrupting the user's current context.

Permission validation is handled by the check_permissions tool, which invokes ComputerUsePermissions.status() to inspect macOS Accessibility and Screen Recording permissions. Error responses include a "permissionNeeded" field when operations fail due to insufficient system access (lines 84-88).

Backward Compatibility with CUA

The server maintains compatibility with legacy "Computer Use Assistant" (CUA) commands through thin wrapper implementations (lines 46-63 and 98-108). Tools like cua_click and cua_screenshot map to the newer snapshot and click implementations, ensuring existing automation scripts continue functioning while the modern MCP schema remains preferred.

Practical Implementation: Python Client Example

To interact with the OpenWork MCP server from a Python application, establish a subprocess connection and exchange JSON-RPC messages:

import json, subprocess

proc = subprocess.Popen(
    ["handsfree-computer-use"], 
    stdin=subprocess.PIPE,
    stdout=subprocess.PIPE, 
    text=True
)

def rpc(method, params=None, req_id=1):
    req = {
        "jsonrpc": "2.0", 
        "id": req_id, 
        "method": method, 
        "params": params or {}
    }
    proc.stdin.write(json.dumps(req) + "\n")
    proc.stdin.flush()
    return json.loads(proc.stdout.readline())

# Initialize connection

print(rpc("initialize"))

# List available tools

tools = rpc("tools/list")["result"]["tools"]
print("Available tools:", [t["name"] for t in tools])

# Capture screen state

snapshot = rpc("tools/call", {
    "name": "snapshot", 
    "arguments": {}
})

# Click a specific coordinate with strict mode

click_result = rpc("tools/call", {
    "name": "click",
    "arguments": {
        "x": 100,
        "y": 200,
        "strict": True
    }
}, req_id=2)

This client demonstrates the complete lifecycle: initialization, capability discovery, and tool execution with safety parameters.

Summary

• OpenWork's MCP integration centers on the MCPServer.swift actor, which implements a JSON-RPC 2.0 server over stdin/stdout streams.
• Method dispatch handles initialize, tools/list, and tools/call requests, routing commands to the appropriate runtime handlers.
• Tool schemas are dynamically generated and exposed via toolSchemas(), providing structured metadata for LLM clients.
• Execution layer delegates to ComputerUseRuntime for high-level UI actions and InputService for low-level input events.
• Security features include strict background mode enforcement and permission checking via ComputerUsePermissions.swift.
• Legacy support maintains backward compatibility with CUA commands while promoting the modern MCP protocol.

Frequently Asked Questions

How does the OpenWork MCP server communicate with client applications?

The server uses standard input and output streams as a transport layer. Client applications spawn the handsfree-computer-use binary as a subprocess and exchange JSON-RPC messages line-by-line through stdin/stdout. This design eliminates network dependencies and allows any programming language capable of process spawning to control macOS UI elements.

What happens when an MCP tool execution fails in OpenWork?

When executeTool(name:args:) encounters an error, the catch block in lines 83-86 invokes errorPayload(_:) to construct a structured error response. This payload includes the original error message and contextual flags such as "permissionNeeded" for accessibility violations or "strictMode" for background enforcement failures. The error is returned as a valid JSON-RPC error object with the original request ID.

Can OpenWork's MCP integration run without granting screen recording permissions?

No, full functionality requires both Accessibility and Screen Recording permissions as verified by ComputerUsePermissions.status(). However, the server remains operational and responds to check_permissions calls to inform clients of the current authorization state. Agents can query this status before attempting snapshot operations that would otherwise fail.

What is the difference between strict mode and normal operation in OpenWork MCP?

Strict mode, enabled via boolean arguments in tool calls, forces all UI interactions to occur in the background without activating target applications. When strict mode is active, the ComputerUseRuntime rejects actions requiring foreground focus and returns an error. This prevents automation workflows from interrupting the user's current workspace context.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}