Ego‑Browser Workflows Explained: Semantic, Visual, and Direct DOM Automation

Ego‑Browser provides three distinct workflows—semantic, visual, and direct DOM/CDP—that agents select based on whether a target page exposes reliable DOM structure, canvas‑based rendering, or requires low‑level browser protocol access.

The ego‑browser automation harness (from the citrolabs/ego-lite repository) gives agents flexible, battle‑tested primitives for interacting with any web page. Rather than forcing a single abstraction onto every site, the runtime exposes three purpose‑built workflows. Each workflow maps to a different layer of the browser stack, letting agents trade convenience for control as the situation demands.

Semantic Workflow: Compact, Human‑Readable Page Automation

The semantic workflow is the default choice for standard websites where the DOM and accessibility tree accurately represent visible content.

How Semantic Mode Works

When you call snapshotText(), the runtime captures a semantic snapshot—a pruned tree of interactive elements with auto‑generated @N references and stable loc= selectors. This produces a compact, text‑based representation that LLMs can reason about efficiently.

From src/helpers.ts, the semantic helpers include:

• snapshotText() – generates the ref map and locator snapshot
• click('@N' | 'loc=...') – resolves refs or locators to elements
• fillInput('@N', value) – types into form fields
• Standard navigation and wait helpers

The Element Resolver (src/element-resolver.ts) handles the translation from refs/locators to concrete CDP element IDs, classifying failures as transient (retryable) or permanent.

Semantic Workflow Example

const task = await useOrCreateTaskSpace('news')
await gotoAndWait('https://example.com', { timeout: 30 })

const snapshot = await snapshotText()
const titles = await js(String.raw`() => {
  const items = [...document.querySelectorAll('h2')]
  return items.map(el => el.innerText)
}`)

cliLog('Found titles:', titles)
await click('@1')                 // click first ref from snapshot
await snapshotText()              // re‑snapshot after navigation

Per the SKILL.md documentation, each snapshotText() call rebuilds the ref map, ensuring @N references stay fresh across navigation or dynamic updates. When a ref expires, helpers fall back to stable locators automatically [SKILL.md#L71‑L84].

Visual Workflow: Coordinate‑Based Control for Canvas Apps

The visual workflow targets pages where the DOM lies—canvas‑heavy editors, virtualized spreadsheets, maps, and design tools where the true editing surface lives in pixels, not markup.

When to Use Visual Mode

Scenario	Why DOM fails	Visual solution
Google Docs/Sheets	Content rendered to <canvas> or off‑screen buffers	Screenshot‑based coordinate interaction
Figma, Miro, Excalidraw	Vector graphics, no semantic element tree	Pixel‑precise click, drag, keyboard
Map interfaces (Google Maps, Mapbox)	Tiles as images, markers overlaid	Viewport coordinate navigation
Virtualized lists (Notion, Asana)	DOM nodes recycled, positions unstable	Screenshot verification + coordinate actions

Visual Workflow Helpers

From src/helpers.ts, visual primitives include:

• captureScreenshot() – viewport image capture
• click([x, y]) – coordinate‑based clicking
• doubleClick([x, y]), moveMouse([x, y]) – gesture simulation
• pressKey(), typeText() – keyboard injection

Visual Workflow Example

await gotoAndWait('https://canvas-app.example.com')
await captureScreenshot()

await click([200, 150])           // start point
await moveMouse([400, 300])      // drag to end point

await captureScreenshot()         // verify the drawing

Because the DOM is unreliable, screenshots become the source of truth. Agents validate actions through visual feedback or export/read‑back checks rather than DOM assertions.

Direct DOM / CDP Workflow: Protocol‑Level Control

The direct DOM / CDP workflow exposes the raw browser for scenarios beyond what semantic or visual helpers cover.

Capabilities of Direct Mode

Use case	Helper	Underlying mechanism
Custom DOM traversal, state manipulation	js(expression)	Runtime.evaluate CDP command
Protocol features (cookies, network, security)	cdp(method, params)	Raw Chrome DevTools Protocol
Bespoke data extraction	js() with return values	Full JavaScript execution context
Performance profiling, request interception	cdp('Network.*'), etc.	Direct CDP domain access

The js() and cdp() helpers are implemented in src/cdp-eval.ts, wrapping Runtime.evaluate and generic protocol methods. The Browser Runtime (src/browser-runtime.ts) manages CDP transport and session lifecycle, providing the conduit for these low‑level calls.

Direct DOM/CDP Example

// Retrieve all cookies via CDP
const cookies = await cdp('Network.getAllCookies')
cliLog('Cookies:', cookies)

// Custom DOM extraction not covered by snapshotText
const shadowData = await js(String.raw`() => {
  const host = document.querySelector('my-component')
  return host?.shadowRoot?.innerHTML
}`)

Combining Workflows in Practice

Real automation tasks mix workflows strategically. A typical pattern:

1. Observe – semantic snapshot or screenshot
2. Act – appropriate workflow helper
3. Verify – follow‑up snapshot, screenshot, or DOM query
4. Report – cliLog(...) for observability

Mixed Workflow Example

await gotoAndWait('https://form.example.com')

// Semantic: fill form using accessible refs
await snapshotText()
await fillInput('@12', 'John Doe')
await click('@15')                // submit button

// Visual: confirm success state appearance
await captureScreenshot()

This flexibility—semantic for structure, visual for rendering, direct DOM/CDP for edge cases—lets agents adapt to any page architecture without abandoning the ego‑browser harness.

Key Source Files

File	Responsibility
src/helpers.ts	Public API surface (snapshotText, click, js, captureScreenshot, etc.)
src/element-resolver.ts	Ref/locator/coordinate → CDP element ID resolution
src/browser-runtime.ts	CDP connection, sessions, event buffering
src/cdp-eval.ts	js() and cdp() helper implementations
skills/ego-browser/SKILL.md	Canonical workflow documentation and usage patterns

Summary

• Semantic workflow uses snapshotText() and @N refs for compact, DOM‑based automation of standard websites.
• Visual workflow uses captureScreenshot() and [x, y] coordinates for canvas apps and virtualized interfaces where DOM structure is unreliable.
• Direct DOM/CDP workflow uses js() and cdp() for raw JavaScript execution and protocol‑level operations.
• Workflows combine freely in single tasks; the runtime in src/browser-runtime.ts ensures consistent CDP session management.
• The Element Resolver in src/element-resolver.ts abstracts away the difference between ref‑based, locator‑based, and coordinate‑based targeting.

Frequently Asked Questions

What happens when a semantic ref like @5 no longer exists?

The helper falls back to the stable loc= selector associated with that element. If both fail, the Element Resolver classifies the error as transient (retry after snapshot refresh) or permanent (element genuinely gone). The ref map rebuilds on every snapshotText() call, so stale refs resolve automatically [SKILL.md#L71‑L84].

Can I use the visual workflow on any page, or only canvas apps?

You can use coordinate‑based actions anywhere, but they are brittle on standard DOM pages where responsive layouts shift element positions. The semantic workflow is preferred for structured pages because refs and locators survive layout changes. Reserve visual mode for sites where DOM traversal is impossible or meaningless.

How does js() differ from running code in snapshotText()?

snapshotText() runs internal scripts to build the accessibility tree and ref map—you don't control this execution. js() is agent‑controlled JavaScript via Runtime.evaluate, able to return arbitrary data, manipulate state, or access browser APIs intentionally excluded from the semantic abstraction. Both route through src/cdp-eval.ts, but js() exposes full execution context to the agent.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}