# How to Automate Browser Interactions with cmux In-App Browser API

> Automate browser interactions using the cmux in-app browser API. Control WKWebView panels programmatically with CLI commands for native headless automation.

- Repository: [manaflow-ai/cmux](https://github.com/manaflow-ai/cmux)
- Tags: how-to-guide
- Published: 2026-03-29

---

**The cmux in-app browser API exposes a JSON-RPC command surface that lets scripts and AI agents programmatically control WKWebView panels via CLI commands like `browser.open_split`, `browser.click`, and `browser.snapshot` for native headless automation within the terminal multiplexer.**

cmux is a terminal-centric macOS window manager that embeds a full-featured browser engine directly into its workspace panes. The **cmux in-app browser API** enables you to automate web interactions—navigation, form filling, screenshots, and JavaScript execution—using stable surface handles and JSON-RPC commands routed through the daemon's V2 protocol in [`Sources/TerminalController.swift`](https://github.com/manaflow-ai/cmux/blob/main/Sources/TerminalController.swift).

## Understanding the Browser Architecture

The browser automation system follows a three-tier architecture: CLI parsing, V2 protocol dispatch, and WebView execution. All commands operate on **surface IDs** (or short refs like `surface:3`) that remain stable even when panes are moved or reorganized.

### CLI to V2 Protocol Pipeline

When you run `cmux browser …` commands, the CLI entry point in [`CLI/cmux.swift`](https://github.com/manaflow-ai/cmux/blob/main/CLI/cmux.swift) parses your input and constructs a JSON-RPC payload. This payload routes to `TerminalController.handleV2Method`, which dispatches to specific browser handlers like `v2BrowserOpenSplit`, `v2BrowserClick`, or `v2BrowserEval` defined in [`Sources/TerminalController.swift`](https://github.com/manaflow-ai/cmux/blob/main/Sources/TerminalController.swift).

### Surface Resolution and Handle Semantics

Every browser command resolves targets through `v2ResolveSurfaceId` (called via `v2ResolveTabManager` → `v2ResolveWorkspace`). The system returns **handle refs** (`window_ref`, `workspace_ref`, `pane_ref`, `surface_ref`) that act as persistent identifiers. According to the agent-browser specification in [`docs/agent-browser-port-spec.md`](https://github.com/manaflow-ai/cmux/blob/main/docs/agent-browser-port-spec.md), these UUID-based handles never change during surface moves, ensuring reliable automation scripts.

The actual WebView interaction occurs in [`Sources/Panels/BrowserPanel.swift`](https://github.com/manaflow-ai/cmux/blob/main/Sources/Panels/BrowserPanel.swift), which hosts the native `WKWebView` and provides methods for navigation, snapshots, and proxy management that the V2 methods consume.

## Core Browser Automation Commands

The browser API implements the agent-browser command surface documented in [`docs/agent-browser-port-spec.md`](https://github.com/manaflow-ai/cmux/blob/main/docs/agent-browser-port-spec.md). Each command accepts a `--surface` parameter and returns JSON containing updated handle refs.

### Surface Identification and Navigation

Before automating, obtain the target surface ID using the identity system:

```bash
cmux identify --json

```

This returns the `surface_id` of the currently focused browser pane. Use this ID to open new splits or navigate:

```bash

# Open a URL in a new split (or reuse the right-most pane)

cmux browser --surface surface:b8d2 open_split url=https://developer.apple.com

# Navigate an existing surface to a new URL

cmux browser --surface surface:4f12 navigate url=https://github.com/manaflow-ai/cmux

```

The `v2BrowserOpenSplit` function (lines 7369+ in [`TerminalController.swift`](https://github.com/manaflow-ai/cmux/blob/main/TerminalController.swift)) handles placement logic, creating splits when necessary or routing to external browsers based on the "external open" branch logic.

### Element Interaction (Click, Type, Fill)

Interact with DOM elements using CSS selectors. The `v2BrowserClick` implementation (line 8157) injects JavaScript to perform the click event:

```bash

# Click an element by CSS selector

cmux browser --surface surface:4f12 click selector="a[href$='README.md']"

# Type text into form fields

cmux browser --surface surface:4f12 type selector="#search" text="cmux automation"

# Fill forms with specific values

cmux browser --surface surface:4f12 fill selector="#username" text="admin"

```

These commands generate JavaScript snippets executed via `BrowserPanel.executeJavaScript`, providing synchronous feedback on success or selector-not-found errors.

### Waiting and Screenshot Capture

Automation scripts often need to pause for page loads or element appearance. The `v2BrowserWait` command polls for selector presence:

```bash

# Wait up to 5000ms for an element to appear

cmux browser --surface surface:4f12 wait selector="#results" timeout=5000

# Capture the visible viewport as PNG

cmux browser --surface surface:4f12 screenshot output=page.png

```

The `v2BrowserSnapshot` function (line 7781) interfaces with `BrowserPanel` to rasterize the `WKWebView` content, returning image data through the JSON-RPC response.

### JavaScript Execution

For complex automation beyond built-in commands, execute arbitrary JavaScript and retrieve results:

```bash
cmux browser --surface surface:4f12 eval script="document.title"

```

Returns:

```json
{ "result": "cmux – a terminal‑centric macOS window manager" }

```

This uses `v2BrowserEval` to pipe scripts through the WebView's JavaScript engine and serialize return values back to the CLI.

## Complete Automation Script Example

The following bash script demonstrates a full workflow: identifying the current surface, opening a search query in a split, interacting with forms, and capturing results:

```bash
#!/usr/bin/env bash
set -euo pipefail

# Identify the currently focused browser surface

SURF=$(cmux identify --json | jq -r '.focused.surface_id')
if [[ -z "$SURF" ]]; then
  echo "No focused browser surface." >&2; exit 1
fi

# Open DuckDuckGo in a new split

cmux browser --surface "$SURF" open_split \
  url="https://duckduckgo.com/?q=cmux+automation"

# Get the new surface ID from the focused pane

NEW_SURF=$(cmux identify --json | jq -r '.focused.surface_id')

# Wait for the search box to render

cmux browser --surface "$NEW_SURF" wait \
  selector="#search_form_input_homepage" timeout=8000

# Type query and submit

cmux browser --surface "$NEW_SURF" type \
  selector="#search_form_input_homepage" text="cmux screenshots"
cmux browser --surface "$NEW_SURF" press key="Enter"

# Wait for results and capture screenshot

cmux browser --surface "$NEW_SURF" wait \
  selector=".result" timeout=5000
cmux browser --surface "$NEW_SURF" screenshot output="cmux-results.png"

# Close the browser surface when done

cmux surface close "$NEW_SURF"

```

This script leverages the stable handle semantics—`surface_id` remains valid throughout the session even if the pane is repositioned within the workspace.

## Key Implementation Files

Understanding the source structure helps when debugging automation failures or extending functionality:

- **[`Sources/TerminalController.swift`](https://github.com/manaflow-ai/cmux/blob/main/Sources/TerminalController.swift)** — Contains V2 protocol dispatch and all browser method implementations (`v2BrowserOpenSplit`, `v2BrowserClick`, `v2BrowserEval`, `v2BrowserSnapshot`, etc.)
- **[`Sources/Panels/BrowserPanel.swift`](https://github.com/manaflow-ai/cmux/blob/main/Sources/Panels/BrowserPanel.swift)** — UI container hosting the `WKWebView`, handling navigation delegates, snapshots, and proxy configuration
- **[`CLI/cmux.swift`](https://github.com/manaflow-ai/cmux/blob/main/CLI/cmux.swift)** — CLI entry point that parses `cmux browser …` arguments and forwards JSON-RPC payloads to the daemon
- **[`docs/agent-browser-port-spec.md`](https://github.com/manaflow-ai/cmux/blob/main/docs/agent-browser-port-spec.md)** — Formal specification of the command surface, method families, and handle semantics
- **[`docs/v2-api-migration.md`](https://github.com/manaflow-ai/cmux/blob/main/docs/v2-api-migration.md)** — Migration guide mapping legacy v1 commands (e.g., `open_browser`) to current `browser.*` syntax
- **[`tests_v2/test_browser_api_comprehensive.py`](https://github.com/manaflow-ai/cmux/blob/main/tests_v2/test_browser_api_comprehensive.py)** — End-to-end test suite serving as reference implementations for correct JSON-RPC payloads

## Summary

- The **cmux in-app browser API** provides native WKWebView automation through a JSON-RPC interface exposed via the CLI.
- All commands flow through **[`TerminalController.swift`](https://github.com/manaflow-ai/cmux/blob/main/TerminalController.swift)** and operate on stable **surface IDs** that persist across workspace reorganizations.
- Available automation primitives include **navigation**, **element interaction** (click/type/fill), **explicit waiting**, **screenshots**, and **arbitrary JavaScript execution**.
- The system supports **headless-style automation** within a visible or backgrounded macOS terminal multiplexer, making it ideal for LLM-driven workflows that require web data extraction.

## Frequently Asked Questions

### How do I obtain the surface ID for browser automation?

Run `cmux identify --json` to retrieve the currently focused surface. The output includes `surface_id` (full UUID) and short refs (e.g., `surface:3`). Use either format with the `--surface` flag in subsequent browser commands. The ID remains stable even if you move the pane to a different workspace or split configuration.

### What is the difference between `browser.open_split` and `browser.navigate`?

`browser.open_split` (implemented in `v2BrowserOpenSplit`) creates a new pane or reuses the right-most pane to open a URL, returning a new `surface_id`. `browser.navigate` (via `v2BrowserNavigate`) changes the URL of an existing browser surface without creating new panes. Use `open_split` when you need side-by-side browser instances; use `navigate` for sequential page loads within the same pane.

### Can I automate cmux browser interactions from Python or Node.js?

Yes. The CLI interface wraps a JSON-RPC protocol over Unix domain sockets. Any language that can spawn subprocesses or open sockets can send payloads matching the schema in [`docs/agent-browser-port-spec.md`](https://github.com/manaflow-ai/cmux/blob/main/docs/agent-browser-port-spec.md). The Python test suite in [`tests_v2/test_browser_api_comprehensive.py`](https://github.com/manaflow-ai/cmux/blob/main/tests_v2/test_browser_api_comprehensive.py) demonstrates the exact JSON structure expected by [`TerminalController.swift`](https://github.com/manaflow-ai/cmux/blob/main/TerminalController.swift).

### Why does my screenshot command return blank images?

Screenshots require the browser surface to have completed rendering. Ensure you call `browser.wait` with a relevant CSS selector or sufficient timeout before `browser.screenshot` to guarantee the `WKWebView` has finished layout and paint operations. The `v2BrowserSnapshot` function captures the current composited layer; if the page is still loading, you may capture an incomplete or transparent buffer.