# How MCP (Model Context Protocol) Integration Works in OpenWork: Architecture and Implementation

> Discover how OpenWork integrates MCP using a Swift JSON-RPC server to connect LLM agents with macOS UI automation. Explore the architecture and implementation details.

- Repository: [Different AI/openwork](https://github.com/different-ai/openwork)
- Tags: architecture
- Published: 2026-07-31

---

**OpenWork implements MCP integration through a Swift-based JSON-RPC server ([`MCPServer.swift`](https://github.com/different-ai/openwork/blob/main/MCPServer.swift)) that bridges LLM agents with macOS UI automation via stdin/stdout communication.**

The OpenWork project leverages the Model Context Protocol (MCP) to enable LLM-driven agents to interact with the macOS operating system through a lightweight, structured interface. At the heart of this MCP integration lies the HandsFree binary, an actor-based server that translates JSON-RPC requests into native UI actions. This architecture allows AI agents to capture screenshots, click UI elements, and inject keyboard input without requiring direct OS bindings.

## MCP Server Architecture and Lifecycle

The core of OpenWork's MCP integration resides in [`packages/handsfree/native/HandsFree/Sources/ComputerUse/MCPServer.swift`](https://github.com/different-ai/openwork/blob/main/packages/handsfree/native/HandsFree/Sources/ComputerUse/MCPServer.swift). This file defines an `actor MCPServer` that manages the entire request-response lifecycle through standard input and output streams.

The server initializes with two primary dependencies:

```swift
actor MCPServer {
    private let runtime = ComputerUseRuntime()
    private let input = InputService()
    // ...
}

```

The `run()` method establishes a continuous loop that reads JSON-RPC messages from stdin:

```swift
func run() async {
    log("Computer Use server starting")
    while let line = readLine(strippingNewline: true) {
        // Parse JSON and extract fields
        let id = json["id"]
        let method = json["method"] as? String ?? ""
        let params = json["params"] as? [String: Any] ?? [:]
        switch method { 
            // Dispatch to appropriate handler
        }
    }
}

```

Empty or malformed lines are ignored with logging, while valid requests are dispatched based on the `method` field. The server echoes the `id` field in all responses to maintain request correlation.

## JSON-RPC Method Handling

The server implements four primary JSON-RPC methods as defined in lines 25-55 of [`MCPServer.swift`](https://github.com/different-ai/openwork/blob/main/MCPServer.swift):

- **`initialize`** (lines 25-32): Returns protocol version, server capabilities, and metadata to establish the connection.
- **`tools/list`** (lines 34-55): Supplies an array of supported tool schemas generated by `toolSchemas()`.
- **`tools/call`** (lines 36-41): Executes a specific tool by name with provided arguments, returning structured results.
- **`notifications/initialized`** (lines 32-34): Currently a no-op reserved for future client-side notification handling.

When an unknown method arrives and the request contains an `id`, the server returns a standard JSON-RPC error with code **-32601** (Method not found) as implemented in lines 42-45.

## Tool Schema Definition and Registration

Each available tool is defined using the `toolSchema(name:description:properties:)` helper function. These schemas conform to JSON Schema specifications and include:

- **name**: The RPC method identifier used in `tools/call` invocations.
- **description**: Human-readable documentation for client UI rendering.
- **inputSchema**: A typed object definition specifying required and optional parameters.

All schemas are aggregated in `toolSchemas()` and returned by the `tools/list` method (lines 49-55). The `snapshot` tool schema exemplifies this structure:

```swift
toolSchema(
    name: "snapshot",
    description: "Return target-window screenshot plus compact semantic AX state …",
    properties: [
        "app": ["type": "string", "description": "Optional running app name …"],
        "pid": ["type": "number", "description": "Optional process id …"],
        // ...
    ]
)

```

## Runtime Execution and UI Automation

The `executeTool(name:args:)` function serves as the dispatcher that maps incoming tool names to concrete macOS automation calls (lines 61-83). This method uses a `switch` statement to route commands to either `ComputerUseRuntime` or `InputService`.

**ComputerUseRuntime** encapsulates high-level UI actions including `snapshot`, `click`, `type`, and `pressKey`. It interfaces directly with macOS Accessibility APIs, CoreGraphics events, and ScreenCaptureKit.

**InputService** provides low-level mouse and keyboard event generation for CUA-compatibility tools.

The execution flow follows this pattern:

```swift
switch name {
case "snapshot":
    let result = try await runtime.snapshot(args: args)
    return jsonResult(id: id, result: result)
case "click":
    let result = try await runtime.click(
        x: intArg(args, "x"),
        y: intArg(args, "y"),
        strict: boolArg(args, "strict")
    )
    return jsonResult(id: id, result: result)
// ...
}

```

All execution branches are wrapped in `do/catch` blocks. Thrown errors are converted to structured payloads via `errorPayload(_:)` (lines 83-86), enriching responses with context such as missing permissions or strict mode violations.

## Security, Permissions, and Strict Mode

OpenWork's MCP integration includes a **strict background mode** that prevents UI actions from activating foreground applications. This safety feature is exposed through boolean parameters in tool schemas and enforced in runtime calls.

When strict mode is enabled via arguments like `"strict": true`, the server rejects actions that would require foreground activation, returning errors containing `"strictMode"` metadata (lines 77-80). This ensures agents can operate without disrupting the user's current context.

Permission validation is handled by the `check_permissions` tool, which invokes `ComputerUsePermissions.status()` to inspect macOS Accessibility and Screen Recording permissions. Error responses include a `"permissionNeeded"` field when operations fail due to insufficient system access (lines 84-88).

## Backward Compatibility with CUA

The server maintains compatibility with legacy "Computer Use Assistant" (CUA) commands through thin wrapper implementations (lines 46-63 and 98-108). Tools like `cua_click` and `cua_screenshot` map to the newer `snapshot` and `click` implementations, ensuring existing automation scripts continue functioning while the modern MCP schema remains preferred.

## Practical Implementation: Python Client Example

To interact with the OpenWork MCP server from a Python application, establish a subprocess connection and exchange JSON-RPC messages:

```python
import json, subprocess

proc = subprocess.Popen(
    ["handsfree-computer-use"], 
    stdin=subprocess.PIPE,
    stdout=subprocess.PIPE, 
    text=True
)

def rpc(method, params=None, req_id=1):
    req = {
        "jsonrpc": "2.0", 
        "id": req_id, 
        "method": method, 
        "params": params or {}
    }
    proc.stdin.write(json.dumps(req) + "\n")
    proc.stdin.flush()
    return json.loads(proc.stdout.readline())

# Initialize connection

print(rpc("initialize"))

# List available tools

tools = rpc("tools/list")["result"]["tools"]
print("Available tools:", [t["name"] for t in tools])

# Capture screen state

snapshot = rpc("tools/call", {
    "name": "snapshot", 
    "arguments": {}
})

# Click a specific coordinate with strict mode

click_result = rpc("tools/call", {
    "name": "click",
    "arguments": {
        "x": 100,
        "y": 200,
        "strict": True
    }
}, req_id=2)

```

This client demonstrates the complete lifecycle: initialization, capability discovery, and tool execution with safety parameters.

## Summary

- **OpenWork's MCP integration** centers on the [`MCPServer.swift`](https://github.com/different-ai/openwork/blob/main/MCPServer.swift) actor, which implements a JSON-RPC 2.0 server over stdin/stdout streams.
- **Method dispatch** handles `initialize`, `tools/list`, and `tools/call` requests, routing commands to the appropriate runtime handlers.
- **Tool schemas** are dynamically generated and exposed via `toolSchemas()`, providing structured metadata for LLM clients.
- **Execution layer** delegates to `ComputerUseRuntime` for high-level UI actions and `InputService` for low-level input events.
- **Security features** include strict background mode enforcement and permission checking via [`ComputerUsePermissions.swift`](https://github.com/different-ai/openwork/blob/main/ComputerUsePermissions.swift).
- **Legacy support** maintains backward compatibility with CUA commands while promoting the modern MCP protocol.

## Frequently Asked Questions

### How does the OpenWork MCP server communicate with client applications?

The server uses standard input and output streams as a transport layer. Client applications spawn the `handsfree-computer-use` binary as a subprocess and exchange JSON-RPC messages line-by-line through stdin/stdout. This design eliminates network dependencies and allows any programming language capable of process spawning to control macOS UI elements.

### What happens when an MCP tool execution fails in OpenWork?

When `executeTool(name:args:)` encounters an error, the catch block in lines 83-86 invokes `errorPayload(_:)` to construct a structured error response. This payload includes the original error message and contextual flags such as `"permissionNeeded"` for accessibility violations or `"strictMode"` for background enforcement failures. The error is returned as a valid JSON-RPC error object with the original request ID.

### Can OpenWork's MCP integration run without granting screen recording permissions?

No, full functionality requires both Accessibility and Screen Recording permissions as verified by `ComputerUsePermissions.status()`. However, the server remains operational and responds to `check_permissions` calls to inform clients of the current authorization state. Agents can query this status before attempting snapshot operations that would otherwise fail.

### What is the difference between strict mode and normal operation in OpenWork MCP?

Strict mode, enabled via boolean arguments in tool calls, forces all UI interactions to occur in the background without activating target applications. When strict mode is active, the `ComputerUseRuntime` rejects actions requiring foreground focus and returns an error. This prevents automation workflows from interrupting the user's current workspace context.