# Apache Mako `computer-use` Package: Browser Automation Capabilities Explained

> Explore Apache Mako's computer-use package for powerful browser automation. Discover how language models interact with UI elements via accessibility, no coordinates needed.

- Repository: [The Apache Software Foundation/maka](https://github.com/apache/maka)
- Tags: deep-dive
- Published: 2026-09-06

---

**The `computer-use` package in Apache Mako provides a full-featured, model-driven browser and desktop automation layer that lets language models discover, observe, and manipulate UI elements through accessibility-based interactions without raw screen coordinates.**

This package exposes a single "computer" tool with comprehensive **browser automation capabilities**, enabling AI agents to control applications programmatically. The implementation spans multiple TypeScript modules in the Mako monorepo, with clear separation between wire protocol schemas, type definitions, session state management, and platform-specific backend dispatch.

## Core Architecture and Tool Schema

The automation surface centers on two definitive files:

- **[`packages/runtime/src/computer-use-tools.ts`](https://github.com/apache/maka/blob/main/packages/runtime/src/computer-use-tools.ts)** — Defines the JSON wire schema (`computerWireParams`), action routing logic, session/frame state machines, and error handling
- **[`packages/runtime/src/computer-use-types.ts`](https://github.com/apache/maka/blob/main/packages/runtime/src/computer-use-types.ts)** — Contains TypeScript type definitions for all actions, observations, and backend contracts

Every tool call is **session-aware**, requiring a `sessionId` and `turnId` pair. The runtime maintains this through `CuaSessionState` ([`cua-session-state.ts`](https://github.com/apache/maka/blob/main/cua-session-state.ts)) and `CuaFrameState` ([`cua-frame-state.ts`](https://github.com/apache/maka/blob/main/cua-frame-state.ts)), which track observation frames, enforce **exact-target validation**, verify **action leases**, and trigger **re-observation** when the UI changes.

## Application Discovery and Launch

### List Running Applications

The `list_apps` capability returns running applications with optional name filtering.

**JSON payload sent by model:**

```json
{
  "action": "list_apps",
  "app": "Safari"
}

```

**Runtime dispatch to backend:**

```typescript
await backend.listApps?.(signal);

```

This calls the `listApps` method of `CuDispatchBackend` as defined in [`computer-use-types.ts`](https://github.com/apache/maka/blob/main/computer-use-types.ts).

### Launch Applications in Background

The `launch_app` action starts an application without stealing foreground focus. This is implemented via the `launchApp` method of the backend, allowing automation workflows to prepare applications before interaction.

## Window Observation and Accessibility Trees

### Capture Structured UI State

The `observe` action (also `capture_observation`) returns an **AX-based accessibility tree** with optional screenshot, optional menu bar capture, and query filtering. This is Mako's primary mechanism for surfacing UI structure to language models.

**Model request:**

```json
{
  "action": "observe",
  "app": "Google Chrome",
  "include_screenshot": true
}

```

**Runtime execution:**

```typescript
await backend.observeApp?.(
  { app: "Google Chrome", includeScreenshot: true },
  signal,
  { sessionId, turnId, toolCallId }
);

```

The `observeApp` and `captureObservation` methods in the backend generate the response, with `registerObservation` in the tools file handling observation caching and frame state updates.

### Screenshot Policy

Screenshots are controlled by `COMPUTER_USE_MODEL_SCREENSHOT_POLICY` in [`computer-use-tools.ts`](https://github.com/apache/maka/blob/main/computer-use-tools.ts). When `include_screenshot: true` is specified, the response includes a base-64 encoded PNG or JPEG.

## Element Interaction Primitives

All element interactions use **accessibility identifiers** rather than raw coordinates, ensuring reproducibility across screen sizes and resolutions. These are defined as `CuSemanticAction` variants in [`computer-use-types.ts`](https://github.com/apache/maka/blob/main/computer-use-types.ts).

### Click Elements

```json
{
  "action": "click_element",
  "observation_id": "obs-123",
  "element_id": "button-ok"
}

```

```typescript
await backend.runSemantic?.(
  {
    type: "click_element",
    observationId: "obs-123",
    elementId: "button-ok"
  },
  signal,
  { sessionId, turnId, toolCallId }
);

```

### Set Values and Type Text

- **`set_value`** — Type a complete string into a field
- **`type`** / **`key`** / **`press_key`** — Send individual keystrokes or strings to the focused element

The `press_key` variant and `runSemantic` dispatcher handle keyboard input routing.

### Select Text Ranges

The `select_text` action enables precise text selection within UI elements, supporting form filling and content extraction workflows.

### Secondary Actions

`secondary_action` invokes context-specific behaviors such as "raise" for window management, mapped to appropriate `CuSemanticAction` variants.

## Scrolling and Viewport Control

The `scroll_element` action supports directional scrolling with page-based units:

```json
{
  "action": "scroll_element",
  "element_id": "scrollable-area-5",
  "scroll_direction": "down",
  "scroll_amount": 10
}

```

**10 units equals one full page**. The `scroll_direction` and `scroll_amount` fields are processed in `runSemantic` with backend dispatch.

## Window Manipulation

The `window_action` capability moves, resizes, or minimizes windows without requiring foreground focus:

```json
{
  "action": "window_action",
  "observation_id": "obs-123",
  "element_id": "window-0",
  "window_action": "move",
  "position": [100, 200]
}

```

```typescript
await backend.runSemantic?.(
  {
    type: "window_action",
    observationId: "obs-123",
    elementId: "window-0",
    action: "move",
    position: { x: 100, y: 200 }
  },
  signal,
  { sessionId, turnId, toolCallId }
);

```

This is validated in [`computer-use-tools.ts`](https://github.com/apache/maka/blob/main/computer-use-tools.ts) and dispatched as a `window_action` variant of `CuSemanticAction`.

## Waiting and Synchronization

The `wait` action pauses execution until UI conditions are met:

```json
{
  "action": "wait",
  "observation_id": "obs-124",
  "wait_for_text": "Save changes?",
  "duration": 10
}

```

**Supported conditions:**
- `wait_for_text` —Pause until specified text appears
- `wait_for_text_gone` — Pause until specified text disappears
- `duration` — Maximum timeout in seconds

The runtime translates this into an internal timer with re-observe loop polling.

## Session Control and Resource Management

The `clear_session` action resets automation state:

- Clears cached observations
- Resets action leases
- Releases backend resources

This calls the `clearSession` method of the tool set, defined in [`computer-use-tools.ts`](https://github.com/apache/maka/blob/main/computer-use-tools.ts).

## Platform Abstraction and Backend Implementation

The **browser automation capabilities** are **platform-agnostic** at the API layer. On macOS:

- The host spawns a signed Swift helper implementing `CuDispatchBackend`
- Communication uses the `maka.cu/2` protocol
- Per-action TCC (Transparency, Consent, and Control) re-checks enforce permissions
- Coordinate authority validation prevents off-screen interactions
- Abort-signal threading enables cancellation

The runtime layer produces `computerWireParams` — the schema that models must satisfy — while OS-independent contracts handle safety and privacy.

## Key Implementation Files

| File | Purpose |
|------|---------|
| [`packages/runtime/src/computer-use-tools.ts`](https://github.com/apache/maka/blob/main/packages/runtime/src/computer-use-tools.ts) | Wire schema, action routing, session/frame state, error handling |
| [`packages/runtime/src/computer-use-types.ts`](https://github.com/apache/maka/blob/main/packages/runtime/src/computer-use-types.ts) | Type definitions for actions, observations, backend contracts |
| [`packages/runtime/src/cua-session-state.ts`](https://github.com/apache/maka/blob/main/packages/runtime/src/cua-session-state.ts) | Session-level state machine (locks, screen-locked, user-intervened) |
| [`packages/runtime/src/cua-frame-state.ts`](https://github.com/apache/maka/blob/main/packages/runtime/src/cua-frame-state.ts) | Frame-level state (observation, action binding, lease validation) |
| [`packages/computer-use/src/computer-use-overlay-hook.ts`](https://github.com/apache/maka/blob/main/packages/computer-use/src/computer-use-overlay-hook.ts) | Presentation overlay (cursor animation, UI feedback) |
| [`docs/computer-use-ui-coverage.md`](https://github.com/apache/maka/blob/main/docs/computer-use-ui-coverage.md) | Exhaustive UI element and action coverage documentation |
| [`apps/desktop/src/main/computer-use-host.ts`](https://github.com/apache/maka/blob/main/apps/desktop/src/main/computer-use-host.ts) | Desktop host wiring backend into Electron rendering process |

## Summary

The Apache Mako `computer-use` package delivers **comprehensive browser automation capabilities** through a well-structured, safety-first architecture:

- **Accessibility-based interaction** — All actions use semantic identifiers, not raw coordinates
- **Session-aware execution** — `sessionId`/`turnId` pairing with lease validation prevents race conditions
- **Platform abstraction** — macOS Swift backend with OS-independent runtime contracts
- **Complete UI surface coverage** — Discovery, observation, interaction, scrolling, window management, and synchronization primitives
- **Model-friendly wire protocol** — JSON schemas in `computerWireParams` with TypeScript type safety

These capabilities enable language models to automate complex desktop and browser workflows while maintaining user privacy and system security.

## Frequently Asked Questions

### What browsers does the `computer-use` package support?

The package supports **any desktop application** with accessibility API exposure, including all major browsers (Chrome, Firefox, Safari, Edge). The `observe` and `runSemantic` actions operate through OS accessibility frameworks rather than browser-specific APIs, making the automation surface universal across macOS applications.

### How does Mako prevent automation from interfering with the user?

Multiple safeguards are implemented: **action leases** prevent stale observations from triggering interactions, **exact-target validation** ensures elements haven't changed since observation, **TCC re-checks** verify permissions on every action, and **session state tracking** detects screen locks or user intervention via `CuaSessionState`. The design prioritizes explicit user consent and observable automation indicators.

### Can the package run without displaying a visible browser window?

Yes. The `launch_app` action starts applications in the background without foreground focus, and `window_action` can manipulate windows without bringing them to the front. However, the accessibility tree still requires the target application to be running; headless operation without any UI process is not supported as the automation relies on AX APIs that need a live application instance.

### What is the relationship between `computer-use` and traditional browser automation tools like Selenium or Playwright?

Unlike Selenium or Playwright, which use WebDriver or DevTools protocols, Mako's **browser automation capabilities** operate at the **OS accessibility layer**. This enables automation of native desktop applications beyond browsers, eliminates coordinate brittleness, and works with unmodified applications. The trade-off is slightly higher latency due to accessibility API overhead and platform-specific backend requirements.