Apache Mako `computer-use` Package: Browser Automation Capabilities Explained

The computer-use package in Apache Mako provides a full-featured, model-driven browser and desktop automation layer that lets language models discover, observe, and manipulate UI elements through accessibility-based interactions without raw screen coordinates.

This package exposes a single "computer" tool with comprehensive browser automation capabilities, enabling AI agents to control applications programmatically. The implementation spans multiple TypeScript modules in the Mako monorepo, with clear separation between wire protocol schemas, type definitions, session state management, and platform-specific backend dispatch.

Core Architecture and Tool Schema

The automation surface centers on two definitive files:

• packages/runtime/src/computer-use-tools.ts — Defines the JSON wire schema (computerWireParams), action routing logic, session/frame state machines, and error handling
• packages/runtime/src/computer-use-types.ts — Contains TypeScript type definitions for all actions, observations, and backend contracts

Every tool call is session-aware, requiring a sessionId and turnId pair. The runtime maintains this through CuaSessionState (cua-session-state.ts) and CuaFrameState (cua-frame-state.ts), which track observation frames, enforce exact-target validation, verify action leases, and trigger re-observation when the UI changes.

Application Discovery and Launch

List Running Applications

The list_apps capability returns running applications with optional name filtering.

JSON payload sent by model:

{
  "action": "list_apps",
  "app": "Safari"
}

Runtime dispatch to backend:

await backend.listApps?.(signal);

This calls the listApps method of CuDispatchBackend as defined in computer-use-types.ts.

Launch Applications in Background

The launch_app action starts an application without stealing foreground focus. This is implemented via the launchApp method of the backend, allowing automation workflows to prepare applications before interaction.

Window Observation and Accessibility Trees

Capture Structured UI State

The observe action (also capture_observation) returns an AX-based accessibility tree with optional screenshot, optional menu bar capture, and query filtering. This is Mako's primary mechanism for surfacing UI structure to language models.

Model request:

{
  "action": "observe",
  "app": "Google Chrome",
  "include_screenshot": true
}

Runtime execution:

await backend.observeApp?.(
  { app: "Google Chrome", includeScreenshot: true },
  signal,
  { sessionId, turnId, toolCallId }
);

The observeApp and captureObservation methods in the backend generate the response, with registerObservation in the tools file handling observation caching and frame state updates.

Screenshot Policy

Screenshots are controlled by COMPUTER_USE_MODEL_SCREENSHOT_POLICY in computer-use-tools.ts. When include_screenshot: true is specified, the response includes a base-64 encoded PNG or JPEG.

Element Interaction Primitives

All element interactions use accessibility identifiers rather than raw coordinates, ensuring reproducibility across screen sizes and resolutions. These are defined as CuSemanticAction variants in computer-use-types.ts.

Click Elements

{
  "action": "click_element",
  "observation_id": "obs-123",
  "element_id": "button-ok"
}

await backend.runSemantic?.(
  {
    type: "click_element",
    observationId: "obs-123",
    elementId: "button-ok"
  },
  signal,
  { sessionId, turnId, toolCallId }
);

Set Values and Type Text

• set_value — Type a complete string into a field
• type / key / press_key — Send individual keystrokes or strings to the focused element

The press_key variant and runSemantic dispatcher handle keyboard input routing.

Select Text Ranges

The select_text action enables precise text selection within UI elements, supporting form filling and content extraction workflows.

Secondary Actions

secondary_action invokes context-specific behaviors such as "raise" for window management, mapped to appropriate CuSemanticAction variants.

Scrolling and Viewport Control

The scroll_element action supports directional scrolling with page-based units:

{
  "action": "scroll_element",
  "element_id": "scrollable-area-5",
  "scroll_direction": "down",
  "scroll_amount": 10
}

10 units equals one full page. The scroll_direction and scroll_amount fields are processed in runSemantic with backend dispatch.

Window Manipulation

The window_action capability moves, resizes, or minimizes windows without requiring foreground focus:

{
  "action": "window_action",
  "observation_id": "obs-123",
  "element_id": "window-0",
  "window_action": "move",
  "position": [100, 200]
}

await backend.runSemantic?.(
  {
    type: "window_action",
    observationId: "obs-123",
    elementId: "window-0",
    action: "move",
    position: { x: 100, y: 200 }
  },
  signal,
  { sessionId, turnId, toolCallId }
);

This is validated in computer-use-tools.ts and dispatched as a window_action variant of CuSemanticAction.

Waiting and Synchronization

The wait action pauses execution until UI conditions are met:

{
  "action": "wait",
  "observation_id": "obs-124",
  "wait_for_text": "Save changes?",
  "duration": 10
}

Supported conditions:

• wait_for_text —Pause until specified text appears
• wait_for_text_gone — Pause until specified text disappears
• duration — Maximum timeout in seconds

The runtime translates this into an internal timer with re-observe loop polling.

Session Control and Resource Management

The clear_session action resets automation state:

• Clears cached observations
• Resets action leases
• Releases backend resources

This calls the clearSession method of the tool set, defined in computer-use-tools.ts.

Platform Abstraction and Backend Implementation

The browser automation capabilities are platform-agnostic at the API layer. On macOS:

• The host spawns a signed Swift helper implementing CuDispatchBackend
• Communication uses the maka.cu/2 protocol
• Per-action TCC (Transparency, Consent, and Control) re-checks enforce permissions
• Coordinate authority validation prevents off-screen interactions
• Abort-signal threading enables cancellation

The runtime layer produces computerWireParams — the schema that models must satisfy — while OS-independent contracts handle safety and privacy.

Key Implementation Files

File	Purpose
packages/runtime/src/computer-use-tools.ts	Wire schema, action routing, session/frame state, error handling
packages/runtime/src/computer-use-types.ts	Type definitions for actions, observations, backend contracts
packages/runtime/src/cua-session-state.ts	Session-level state machine (locks, screen-locked, user-intervened)
packages/runtime/src/cua-frame-state.ts	Frame-level state (observation, action binding, lease validation)
packages/computer-use/src/computer-use-overlay-hook.ts	Presentation overlay (cursor animation, UI feedback)
docs/computer-use-ui-coverage.md	Exhaustive UI element and action coverage documentation
apps/desktop/src/main/computer-use-host.ts	Desktop host wiring backend into Electron rendering process

Summary

The Apache Mako computer-use package delivers comprehensive browser automation capabilities through a well-structured, safety-first architecture:

• Accessibility-based interaction — All actions use semantic identifiers, not raw coordinates
• Session-aware execution — sessionId/turnId pairing with lease validation prevents race conditions
• Platform abstraction — macOS Swift backend with OS-independent runtime contracts
• Complete UI surface coverage — Discovery, observation, interaction, scrolling, window management, and synchronization primitives
• Model-friendly wire protocol — JSON schemas in computerWireParams with TypeScript type safety

These capabilities enable language models to automate complex desktop and browser workflows while maintaining user privacy and system security.

Frequently Asked Questions

What browsers does the computer-use package support?

The package supports any desktop application with accessibility API exposure, including all major browsers (Chrome, Firefox, Safari, Edge). The observe and runSemantic actions operate through OS accessibility frameworks rather than browser-specific APIs, making the automation surface universal across macOS applications.

How does Mako prevent automation from interfering with the user?

Multiple safeguards are implemented: action leases prevent stale observations from triggering interactions, exact-target validation ensures elements haven't changed since observation, TCC re-checks verify permissions on every action, and session state tracking detects screen locks or user intervention via CuaSessionState. The design prioritizes explicit user consent and observable automation indicators.

Can the package run without displaying a visible browser window?

Yes. The launch_app action starts applications in the background without foreground focus, and window_action can manipulate windows without bringing them to the front. However, the accessibility tree still requires the target application to be running; headless operation without any UI process is not supported as the automation relies on AX APIs that need a live application instance.

What is the relationship between computer-use and traditional browser automation tools like Selenium or Playwright?

Unlike Selenium or Playwright, which use WebDriver or DevTools protocols, Mako's browser automation capabilities operate at the OS accessibility layer. This enables automation of native desktop applications beyond browsers, eliminates coordinate brittleness, and works with unmodified applications. The trade-off is slightly higher latency due to accessibility API overhead and platform-specific backend requirements.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}