# How pi-computer-use Integrates Browser Pages via Chrome DevTools Protocol (CDP) as UI Roots

> Learn how pi-computer-use uses Chrome DevTools Protocol CDP to integrate browser tabs as UI roots for automated interaction bypassing accessibility APIs.

- Repository: [injaneity/pi-computer-use](https://github.com/injaneity/pi-computer-use)
- Tags: how-to-guide
- Published: 2026-07-16

---

**pi-computer-use treats Chromium browser tabs as first-class UI roots by connecting through the Chrome DevTools Protocol (CDP) when the `PI_COMPUTER_USE_CDP_PORT` environment variable is set, enabling automated interaction without platform-specific accessibility APIs.**

The `pi-computer-use` framework extends its UI automation capabilities beyond native macOS and Windows applications to include Chromium-family browsers. By leveraging the **Chrome DevTools Protocol (CDP)**, the system treats browser pages as interactive roots within the same architectural model used for platform accessibility trees, allowing seamless automation of web applications alongside native desktop software.

## Enabling CDP Mode

CDP integration is optional and activates when the environment variable `PI_COMPUTER_USE_CDP_PORT` specifies a valid port number. The function `cdpEnabled()` in [`src/cdp.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/cdp.ts) validates this configuration and checks for WebSocket API availability. If these conditions are not met, the CDP module remains inert and the system falls back to native accessibility backends.

To enable CDP support, launch Chromium with the `--remote-debugging-port` flag and set the environment variable:

```bash

# Launch Chrome with remote debugging

/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --remote-debugging-port=9222

# In your pi-computer-use environment

export PI_COMPUTER_USE_CDP_PORT=9222

```

## Discovering Browser Pages

Once enabled, the system discovers available tabs through the standard CDP HTTP endpoint. The `cdpPages()` function queries `http://127.0.0.1:<port>/json/list` to retrieve all debuggable targets, filtering for local pages that expose a `webSocketDebuggerUrl`.

The `listCdpPageContexts()` function then transforms these targets into `CdpPageContext` objects, assigning each a `contextId` in the format `browser:<targetId>`. This identifier allows the rest of the system to reference specific browser tabs using the same addressing scheme as native UI elements.

## Connecting to Target Tabs

To establish control over a specific browser window, `cdpTabForWindow(windowTitle, frame)` manages tab selection through the `connectedTabs` cache. The selection logic follows a specific disambiguation order implemented in `pickTab()`:

1. **Exact title match** – Tab title equals the target window title
2. **Prefix match** – Tab title starts with the window title
3. **Frame matching** – Optional screen coordinate intersection checks
4. **Visibility heuristics** – Selects the visible tab (`document.visibilityState === "visible"`)

Once identified, `CdpTab.connect(wsUrl, targetId, title)` opens a WebSocket connection to the tab's debugger URL and enables the `Runtime` and `Page` CDP domains. This connection persists for the duration of the automation session, buffering console messages and exceptions for later retrieval.

## Capturing Page State and Accessibility

When the system requests a UI snapshot via `cdpSnapshotForContext(contextId)`, the `CdpTab` instance executes a multi-step capture process:

- **Text extraction** – Evaluates `document.body.innerText` to capture readable content
- **Accessibility tree retrieval** – Calls `Accessibility.getFullAXTree` to obtain the complete browser accessibility structure
- **Target generation** – Processes AX nodes through `cdpSnapshotOutline()` to create `CdpSnapshotTarget` objects for actionable elements like buttons, links, and text inputs

The resulting snapshot merges with the system's internal model, exposing browser elements through the same interface as native macOS Accessibility or Windows UI Automation nodes.

## Handling Browser Console and Errors

The `CdpTab.handleMessage()` method listens for `Runtime.consoleAPICalled` and `Runtime.exceptionThrown` events, maintaining a circular buffer of recent messages limited by `CONSOLE_BUFFER_LIMIT`. This buffer provides diagnostic parity with native backends through the `drainConsole()` method, which flushes accumulated logs and errors for inclusion in tool results.

## Executing Interactions via CDP

Interaction functions translate high-level commands into CDP Input domain methods. The system implements specific handlers such as `cdpClickForContext`, `cdpTypeForContext`, `cdpScrollForContext`, and `cdpKeypressForContext`, which invoke:

- `Input.dispatchMouseEvent` for clicks and scrolls
- `Input.insertText` and `Input.dispatchKeyEvent` for typing
- `Runtime.evaluate` for JavaScript execution

These functions target specific backend node IDs or perform actions on the currently focused element, maintaining consistency with the native automation API surface.

## Unified Root Architecture

The integration bridges CDP-derived browser trees with native accessibility backends through [`src/runtime.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/runtime.ts), which orchestrates platform-specific handlers. When CDP is enabled, browser pages appear alongside macOS and Windows UI elements as additional roots. The [`src/bridge.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/bridge.ts) module routes actions to CDP functions when encountering contexts with `source` equal to `"browser_ax"`, while [`src/platform/macos/browser.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/platform/macos/browser.ts) handles macOS-specific delegation logic for Chrome and Edge instances.

## Code Examples

Enable CDP and connect to a specific browser tab:

```typescript
// Enable CDP by setting the env var to the browser's remote debugging port
process.env.PI_COMPUTER_USE_CDP_PORT = "9222";

// Get a CDP tab that matches the active window title "My Web App"
const tab = await cdpTabForWindow("My Web App");
if (tab) {
  // Navigate to a URL
  await tab.navigate("https://example.com");

  // Click a button identified by its backend node id
  await tab.clickBackendNode(12345);

  // Type into a focused element
  await tab.typeIntoFocused("Hello world");

  // Retrieve recent console messages
  const logs = tab.drainConsole();
  console.log(logs);
}

```

Capture a snapshot of the current page context:

```typescript
// Obtain a snapshot of the current page as a pi-computer-use context
const ctxId = "browser:abcdef1234"; // contextId of a previously discovered tab
const snapshot = await cdpSnapshotForContext(ctxId);
if (snapshot) {
  console.log("Page title:", snapshot.title);
  console.log("Text content length:", snapshot.text.length);
  console.log("Interactive targets:", snapshot.targets.length);
}

```

## Summary

- **Optional activation** – CDP integration requires setting `PI_COMPUTER_USE_CDP_PORT` and launching Chromium with `--remote-debugging-port`
- **Page discovery** – [`src/cdp.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/cdp.ts) queries `http://127.0.0.1:<port>/json/list` and maps targets to `browser:<targetId>` contexts
- **Smart tab selection** – `pickTab()` uses title matching, frame intersection, and visibility heuristics to identify the correct tab
- **Unified model** – Browser accessibility trees merge with native OS trees through `CdpSnapshotTarget` objects and `"browser_ax"` source tagging
- **Full interaction suite** – Click, type, scroll, and keypress actions map to CDP `Input` domain commands
- **Diagnostic parity** – Console and exception buffering via `drainConsole()` provides equivalent debugging to native backends

## Frequently Asked Questions

### How does pi-computer-use identify the correct browser tab among multiple open pages?

The system uses `cdpTabForWindow()` to match the target window title against available tabs, falling back to `pickTab()` which applies a priority order: exact title matches, prefix matches, optional frame-based coordinate intersection, and finally visibility state (`document.visibilityState === "visible"`). This ensures reliable selection even when multiple similar pages are open.

### What CDP domains does pi-computer-use require to function?

The implementation utilizes several Chrome DevTools Protocol domains: `Runtime` for JavaScript evaluation and console monitoring, `Page` for navigation events, `Accessibility` for retrieving the AX tree via `getFullAXTree`, and `Input` for dispatching mouse events, text insertion, and keyboard actions. The `CdpTab.connect()` method automatically enables the necessary domains upon connection.

### Can pi-computer-use access browser console output when using CDP mode?

Yes. The `CdpTab` class maintains an internal buffer of console messages and exceptions captured through `Runtime.consoleAPICalled` and `Runtime.exceptionThrown` events. Calling `drainConsole()` retrieves and clears this buffer, attaching diagnostic output to tool results in the same manner as native platform backends.

### How does the CDP integration differ from native macOS or Windows browser automation?

Unlike platform-specific accessibility APIs that rely on OS-level hooks, the CDP approach communicates directly with the browser via WebSocket, eliminating dependencies on AppleScript or Windows UI Automation for Chromium-based browsers. This allows `pi-computer-use` to treat browser pages as roots within the same abstract model used for native applications, as coordinated through [`src/bridge.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/bridge.ts) and [`src/runtime.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/runtime.ts).