How Apache Maka Implements End-to-End Computer Use Capability

Apache Maka enables large language models to perceive and control macOS desktops through a strictly validated pipeline that translates high-level model requests into native accessibility actions via a signed Swift helper binary.

Apache Maka provides a sophisticated computer use capability that bridges AI models with local desktop environments. This system transforms natural language intentions into concrete macOS accessibility operations through a multi-layered architecture involving Zod schema validation, JSON-RPC process management, and runtime state tracking.

Four-Layer Architecture of the Computer Use System

The implementation in apache/maka organizes functionality into four distinct layers that handle validation, backend selection, protocol communication, and runtime orchestration.

1. Model-Facing Tool Schema

The entry point for model interaction is the computerWireParams schema defined in packages/runtime/src/computer-use-tools.ts. This strict Zod schema validates every JSON payload the model sends, defining the complete action set—including list_apps, observe, click_element, type_text, and scroll—along with their required and optional fields. This validation layer ensures that only well-formed requests reach the host system.

2. Backend Selection Logic

The selectComputerUseBackend function in packages/computer-use/src/select-backend.ts determines which native driver to instantiate. Currently, this capability is restricted to macOS, where it constructs a CuDispatchBackend targeting the maka-cu binary. On non-macOS platforms, the function disables the capability entirely. This layer returns both the backend instance and the set of ComputerUseTool objects exposed to the runtime.

3. Host-Side Service and JSON-RPC Protocol

The MakaCuService class in packages/computer-use/src/maka-cu-service.ts manages the lifecycle of the native helper. It spawns the maka-cu binary, maintains a JSON-RPC 2.0 dialogue over stdio, and executes a mandatory handshake (host.hello) defined in packages/computer-use/src/maka-cu-protocol.ts. The service handles restart logic, backoff strategies, and timeout enforcement while translating raw RPC responses into typed MakaCuEnvelope objects or surface-level MakaCuRpcError instances.

4. Runtime Tool Orchestration

The buildComputerUseTools function in packages/runtime/src/computer-use-tools.ts binds the backend to the active session state. It tracks observations across CuaFrameState, enforces session-level locks (screen-locked, user-intervened, blocked-URL), and manages action fingerprinting. This layer generates the final ComputerToolResult containing the text summary, optional screenshot, and error codes for the model to consume.

End-to-End Execution Flow

When a model invokes the computer use capability, the system executes an eight-step pipeline:

1. Model invocation – The runtime receives a computer function call (e.g., action:"observe") and validates arguments against computerWireParams.

2. Backend dispatch – buildComputerUseTools forwards the request to the CuDispatchBackend selected during initialization.

3. Service request – MakaCuService.request constructs a JSON-RPC 2.0 request with a unique ID, method name, and parameters, writing it to the child process's stdin.

4. Handshake verification – On first start, ensureStarted executes host.hello and validates the MakaCuHandshake response, which confirms executor capabilities, limits, and protocol version compatibility.

5. Native execution – The maka-cu executable (a signed Swift binary) performs concrete macOS accessibility actions, including window enumeration, AX element inspection, cursor movement, and key event injection.

6. Response parsing – The readEnvelope helper in maka-cu-protocol.ts validates the line-delimited JSON-RPC response against closed-set fields, producing typed envelopes or throwing MakaCuProtocolViolation for malformed data.

7. State synchronization – The runtime updates CuaSessionState, determines if re-observation is required via shouldReobserveAfter, and generates the human-readable summary through renderObservationForModel.

8. Result emission – The system returns a ComputerToolResult containing the observation text, base64-encoded screenshot (when policy allows), and a closed-set error code from @maka/core/computer-use if the action failed.

All error codes map to specific recovery strategies—for example, target_missing triggers re-observation logic—ensuring the model receives actionable feedback rather than raw system errors.

Implementation Examples

Selecting the Backend on macOS

import { selectComputerUseBackend } from '@maka/computer-use';

const backend = selectComputerUseBackend({
  binaryPath: '/usr/local/bin/maka-cu',
  expectedBinarySha256: 'a3b2c4…',          // SHA-256 of the signed helper
  compressFrame: (b64, mime) => ({        // optional image compression
    base64: b64,
    mimeType: 'image/png',
  }),
});

Source: packages/computer-use/src/select-backend.ts

Initializing the MakaCuService

import { MakaCuService } from '@maka/computer-use';

const service = new MakaCuService({
  binaryPath: '/usr/local/bin/maka-cu',
  imageDir: '/tmp/maka-cu-images',
  hostVersion: '0.1.0',
});
await service.ensureStarted();        // runs `host.hello` and validates the handshake

Source: packages/computer-use/src/maka-cu-service.ts

Executing Model Actions

import { computer } from '@maka/runtime';   // the function exposed to the model

// List all running apps (model-facing)
const list = await computer({
  action: 'list_apps',
});

// Observe a specific app window
const obs = await computer({
  action: 'observe',
  app: 'TextEdit',
  include_screenshot: true,
});

// Click an element obtained from the observation
await computer({
  action: 'click_element',
  observation_id: obs.observation_id,
  element_id: obs.elements[0].element_id,
});

Source: packages/runtime/src/computer-use-tools.ts

Handling Target Missing Errors

try {
  await computer({ action: 'click_element', ... });
} catch (e) {
  if (e.error === 'target_missing') {
    // The element disappeared – get a fresh observation first
    const fresh = await computer({ action: 'observe', app: 'TextEdit' });
    await computer({
      action: 'click_element',
      observation_id: fresh.observation_id,
      element_id: fresh.elements[0].element_id,
    });
  }
}

Source: Error-mapping logic in buildComputerUseTools (SESSION_BLOCK_RECOVERY, BINDING_FAILURE_RECOVERY).

Key Source Files

File	Purpose
packages/computer-use/src/select-backend.ts	Chooses and constructs the CuDispatchBackend and builds the tool set for the runtime.
packages/computer-use/src/maka-cu-service.ts	Manages the maka-cu child process lifecycle, handshake, JSON-RPC request/response, and restart/back-off logic.
packages/computer-use/src/maka-cu-protocol.ts	Parses and validates JSON-RPC envelopes, defines protocol constants (MAKA_CU_PROTOCOL_VERSION), and provides type guards.
packages/runtime/src/computer-use-tools.ts	Implements the model-facing computer tool, validates arguments via computerWireParams, and manages session state.
packages/runtime/src/computer-use-types.ts	Type definitions for backend interaction including CuDispatchBackend, CuObservation, and CuRunResult.

Summary

• Apache Maka's computer use capability is currently macOS-only, utilizing a signed Swift binary (maka-cu) to execute accessibility commands.
• Strict validation occurs at multiple levels: computerWireParams for model input, readEnvelope for RPC responses, and closed-set error codes for failure modes.
• Robust process management via MakaCuService includes automatic restart logic, cancellation support ($/cancel), and handshake verification.
• Session-aware orchestration tracks UI state through CuaFrameState and CuaSessionState, enabling intelligent re-observation when elements become stale.
• Model-friendly error recovery maps technical failures (e.g., target_missing) to actionable recovery instructions rather than exposing raw system exceptions.

Frequently Asked Questions

What platforms support Apache Maka's computer use capability?

Currently, the computer use capability is exclusively supported on macOS. The selectComputerUseBackend function explicitly checks the operating system and only constructs a CuDispatchBackend for macOS, returning a disabled state on other platforms. This limitation exists because the maka-cu binary relies on macOS-specific accessibility APIs (AX) and Swift-based system integrations.

How does the computerWireParams schema validate model requests?

The computerWireParams schema in packages/runtime/src/computer-use-tools.ts uses Zod to enforce strict typing on all incoming model requests. It validates that required fields like action match specific string literals (list_apps, observe, click_element, etc.) and that parameters such as element_id or observation_id follow expected formats. This prevents malformed or potentially dangerous commands from reaching the native execution layer.

How does the maka-cu service handle process crashes or hangs?

The MakaCuService class implements comprehensive lifecycle management including restart budgets, exponential backoff, and timeout enforcement. If the maka-cu binary crashes or becomes unresponsive, the ensureStarted method automatically respawns the process up to a configured limit, re-establishes the JSON-RPC connection, and re-executes the host.hello handshake to verify protocol compatibility before accepting new commands.

What happens when a UI element disappears between observation and interaction?

When an action like click_element fails because the target no longer exists, the system returns a target_missing error code from the closed-set defined in @maka/core/computer-use. The buildComputerUseTools logic includes recovery handlers that prompt the model to re-observe the application state via shouldReobserveAfter, obtain fresh CuaFrameState, and select a new valid element ID before retrying the interaction.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}