# How Apache Maka Implements End-to-End Computer Use Capability

> Discover how Apache Maka grants LLMs desktop control. Learn about its validated pipeline translating requests to native accessibility actions via Swift.

- Repository: [The Apache Software Foundation/maka](https://github.com/apache/maka)
- Tags: deep-dive
- Published: 2026-09-02

---

**Apache Maka enables large language models to perceive and control macOS desktops through a strictly validated pipeline that translates high-level model requests into native accessibility actions via a signed Swift helper binary.**

Apache Maka provides a sophisticated **computer use capability** that bridges AI models with local desktop environments. This system transforms natural language intentions into concrete macOS accessibility operations through a multi-layered architecture involving Zod schema validation, JSON-RPC process management, and runtime state tracking.

## Four-Layer Architecture of the Computer Use System

The implementation in `apache/maka` organizes functionality into four distinct layers that handle validation, backend selection, protocol communication, and runtime orchestration.

### 1. Model-Facing Tool Schema

The entry point for model interaction is the **`computerWireParams`** schema defined in [`packages/runtime/src/computer-use-tools.ts`](https://github.com/apache/maka/blob/main/packages/runtime/src/computer-use-tools.ts). This strict Zod schema validates every JSON payload the model sends, defining the complete action set—including `list_apps`, `observe`, `click_element`, `type_text`, and `scroll`—along with their required and optional fields. This validation layer ensures that only well-formed requests reach the host system.

### 2. Backend Selection Logic

The **`selectComputerUseBackend`** function in [`packages/computer-use/src/select-backend.ts`](https://github.com/apache/maka/blob/main/packages/computer-use/src/select-backend.ts) determines which native driver to instantiate. Currently, this capability is restricted to macOS, where it constructs a **`CuDispatchBackend`** targeting the `maka-cu` binary. On non-macOS platforms, the function disables the capability entirely. This layer returns both the backend instance and the set of **`ComputerUseTool`** objects exposed to the runtime.

### 3. Host-Side Service and JSON-RPC Protocol

The **`MakaCuService`** class in [`packages/computer-use/src/maka-cu-service.ts`](https://github.com/apache/maka/blob/main/packages/computer-use/src/maka-cu-service.ts) manages the lifecycle of the native helper. It spawns the `maka-cu` binary, maintains a JSON-RPC 2.0 dialogue over stdio, and executes a mandatory handshake (`host.hello`) defined in [`packages/computer-use/src/maka-cu-protocol.ts`](https://github.com/apache/maka/blob/main/packages/computer-use/src/maka-cu-protocol.ts). The service handles restart logic, backoff strategies, and timeout enforcement while translating raw RPC responses into typed **`MakaCuEnvelope`** objects or surface-level **`MakaCuRpcError`** instances.

### 4. Runtime Tool Orchestration

The **`buildComputerUseTools`** function in [`packages/runtime/src/computer-use-tools.ts`](https://github.com/apache/maka/blob/main/packages/runtime/src/computer-use-tools.ts) binds the backend to the active session state. It tracks observations across **`CuaFrameState`**, enforces session-level locks (screen-locked, user-intervened, blocked-URL), and manages action fingerprinting. This layer generates the final **`ComputerToolResult`** containing the text summary, optional screenshot, and error codes for the model to consume.

## End-to-End Execution Flow

When a model invokes the computer use capability, the system executes an eight-step pipeline:

1. **Model invocation** – The runtime receives a `computer` function call (e.g., `action:"observe"`) and validates arguments against `computerWireParams`.

2. **Backend dispatch** – `buildComputerUseTools` forwards the request to the **`CuDispatchBackend`** selected during initialization.

3. **Service request** – `MakaCuService.request` constructs a JSON-RPC 2.0 request with a unique ID, method name, and parameters, writing it to the child process's stdin.

4. **Handshake verification** – On first start, `ensureStarted` executes `host.hello` and validates the **`MakaCuHandshake`** response, which confirms executor capabilities, limits, and protocol version compatibility.

5. **Native execution** – The `maka-cu` executable (a signed Swift binary) performs concrete macOS accessibility actions, including window enumeration, AX element inspection, cursor movement, and key event injection.

6. **Response parsing** – The `readEnvelope` helper in [`maka-cu-protocol.ts`](https://github.com/apache/maka/blob/main/maka-cu-protocol.ts) validates the line-delimited JSON-RPC response against closed-set fields, producing typed envelopes or throwing `MakaCuProtocolViolation` for malformed data.

7. **State synchronization** – The runtime updates **`CuaSessionState`**, determines if re-observation is required via `shouldReobserveAfter`, and generates the human-readable summary through `renderObservationForModel`.

8. **Result emission** – The system returns a `ComputerToolResult` containing the observation text, base64-encoded screenshot (when policy allows), and a closed-set error code from `@maka/core/computer-use` if the action failed.

All error codes map to specific recovery strategies—for example, `target_missing` triggers re-observation logic—ensuring the model receives actionable feedback rather than raw system errors.

## Implementation Examples

### Selecting the Backend on macOS

```typescript
import { selectComputerUseBackend } from '@maka/computer-use';

const backend = selectComputerUseBackend({
  binaryPath: '/usr/local/bin/maka-cu',
  expectedBinarySha256: 'a3b2c4…',          // SHA-256 of the signed helper
  compressFrame: (b64, mime) => ({        // optional image compression
    base64: b64,
    mimeType: 'image/png',
  }),
});

```

*Source:* [`packages/computer-use/src/select-backend.ts`](https://github.com/apache/maka/blob/main/packages/computer-use/src/select-backend.ts)

### Initializing the MakaCuService

```typescript
import { MakaCuService } from '@maka/computer-use';

const service = new MakaCuService({
  binaryPath: '/usr/local/bin/maka-cu',
  imageDir: '/tmp/maka-cu-images',
  hostVersion: '0.1.0',
});
await service.ensureStarted();        // runs `host.hello` and validates the handshake

```

*Source:* [`packages/computer-use/src/maka-cu-service.ts`](https://github.com/apache/maka/blob/main/packages/computer-use/src/maka-cu-service.ts)

### Executing Model Actions

```typescript
import { computer } from '@maka/runtime';   // the function exposed to the model

// List all running apps (model-facing)
const list = await computer({
  action: 'list_apps',
});

// Observe a specific app window
const obs = await computer({
  action: 'observe',
  app: 'TextEdit',
  include_screenshot: true,
});

// Click an element obtained from the observation
await computer({
  action: 'click_element',
  observation_id: obs.observation_id,
  element_id: obs.elements[0].element_id,
});

```

*Source:* [`packages/runtime/src/computer-use-tools.ts`](https://github.com/apache/maka/blob/main/packages/runtime/src/computer-use-tools.ts)

### Handling Target Missing Errors

```typescript
try {
  await computer({ action: 'click_element', ... });
} catch (e) {
  if (e.error === 'target_missing') {
    // The element disappeared – get a fresh observation first
    const fresh = await computer({ action: 'observe', app: 'TextEdit' });
    await computer({
      action: 'click_element',
      observation_id: fresh.observation_id,
      element_id: fresh.elements[0].element_id,
    });
  }
}

```

*Source:* Error-mapping logic in `buildComputerUseTools` (`SESSION_BLOCK_RECOVERY`, `BINDING_FAILURE_RECOVERY`).

## Key Source Files

| File | Purpose |
|------|---------|
| [`packages/computer-use/src/select-backend.ts`](https://github.com/apache/maka/blob/main/packages/computer-use/src/select-backend.ts) | Chooses and constructs the `CuDispatchBackend` and builds the tool set for the runtime. |
| [`packages/computer-use/src/maka-cu-service.ts`](https://github.com/apache/maka/blob/main/packages/computer-use/src/maka-cu-service.ts) | Manages the `maka-cu` child process lifecycle, handshake, JSON-RPC request/response, and restart/back-off logic. |
| [`packages/computer-use/src/maka-cu-protocol.ts`](https://github.com/apache/maka/blob/main/packages/computer-use/src/maka-cu-protocol.ts) | Parses and validates JSON-RPC envelopes, defines protocol constants (`MAKA_CU_PROTOCOL_VERSION`), and provides type guards. |
| [`packages/runtime/src/computer-use-tools.ts`](https://github.com/apache/maka/blob/main/packages/runtime/src/computer-use-tools.ts) | Implements the model-facing `computer` tool, validates arguments via `computerWireParams`, and manages session state. |
| [`packages/runtime/src/computer-use-types.ts`](https://github.com/apache/maka/blob/main/packages/runtime/src/computer-use-types.ts) | Type definitions for backend interaction including `CuDispatchBackend`, `CuObservation`, and `CuRunResult`. |

## Summary

- **Apache Maka's computer use capability** is currently macOS-only, utilizing a signed Swift binary (`maka-cu`) to execute accessibility commands.
- **Strict validation** occurs at multiple levels: `computerWireParams` for model input, `readEnvelope` for RPC responses, and closed-set error codes for failure modes.
- **Robust process management** via `MakaCuService` includes automatic restart logic, cancellation support (`$/cancel`), and handshake verification.
- **Session-aware orchestration** tracks UI state through `CuaFrameState` and `CuaSessionState`, enabling intelligent re-observation when elements become stale.
- **Model-friendly error recovery** maps technical failures (e.g., `target_missing`) to actionable recovery instructions rather than exposing raw system exceptions.

## Frequently Asked Questions

### What platforms support Apache Maka's computer use capability?

Currently, the computer use capability is exclusively supported on macOS. The `selectComputerUseBackend` function explicitly checks the operating system and only constructs a `CuDispatchBackend` for macOS, returning a disabled state on other platforms. This limitation exists because the `maka-cu` binary relies on macOS-specific accessibility APIs (AX) and Swift-based system integrations.

### How does the computerWireParams schema validate model requests?

The `computerWireParams` schema in [`packages/runtime/src/computer-use-tools.ts`](https://github.com/apache/maka/blob/main/packages/runtime/src/computer-use-tools.ts) uses Zod to enforce strict typing on all incoming model requests. It validates that required fields like `action` match specific string literals (`list_apps`, `observe`, `click_element`, etc.) and that parameters such as `element_id` or `observation_id` follow expected formats. This prevents malformed or potentially dangerous commands from reaching the native execution layer.

### How does the maka-cu service handle process crashes or hangs?

The `MakaCuService` class implements comprehensive lifecycle management including restart budgets, exponential backoff, and timeout enforcement. If the `maka-cu` binary crashes or becomes unresponsive, the `ensureStarted` method automatically respawns the process up to a configured limit, re-establishes the JSON-RPC connection, and re-executes the `host.hello` handshake to verify protocol compatibility before accepting new commands.

### What happens when a UI element disappears between observation and interaction?

When an action like `click_element` fails because the target no longer exists, the system returns a `target_missing` error code from the closed-set defined in `@maka/core/computer-use`. The `buildComputerUseTools` logic includes recovery handlers that prompt the model to re-observe the application state via `shouldReobserveAfter`, obtain fresh `CuaFrameState`, and select a new valid element ID before retrying the interaction.