# Ego‑Browser Workflows Explained: Semantic, Visual, and Direct DOM Automation

> Explore Ego-Browser workflows: semantic, visual, and direct DOM automation. Choose the right approach for reliable DOM, canvas rendering, or low-level browser access with Ego-Lite.

- Repository: [CitroLabs/ego-lite](https://github.com/citrolabs/ego-lite)
- Tags: deep-dive
- Published: 2026-08-15

---

**Ego‑Browser provides three distinct workflows—semantic, visual, and direct DOM/CDP—that agents select based on whether a target page exposes reliable DOM structure, canvas‑based rendering, or requires low‑level browser protocol access.**

The **ego‑browser** automation harness (from the `citrolabs/ego-lite` repository) gives agents flexible, battle‑tested primitives for interacting with any web page. Rather than forcing a single abstraction onto every site, the runtime exposes three purpose‑built workflows. Each workflow maps to a different layer of the browser stack, letting agents trade convenience for control as the situation demands.

## Semantic Workflow: Compact, Human‑Readable Page Automation

The **semantic workflow** is the default choice for standard websites where the DOM and accessibility tree accurately represent visible content.

### How Semantic Mode Works

When you call `snapshotText()`, the runtime captures a **semantic snapshot**—a pruned tree of interactive elements with auto‑generated `@N` references and stable `loc=` selectors. This produces a compact, text‑based representation that LLMs can reason about efficiently.

From [`src/helpers.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/helpers.ts), the semantic helpers include:

- `snapshotText()` – generates the ref map and locator snapshot
- `click('@N' | 'loc=...')` – resolves refs or locators to elements
- `fillInput('@N', value)` – types into form fields
- Standard navigation and wait helpers

The **Element Resolver** ([`src/element-resolver.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/element-resolver.ts)) handles the translation from refs/locators to concrete CDP element IDs, classifying failures as transient (retryable) or permanent.

### Semantic Workflow Example

```javascript
const task = await useOrCreateTaskSpace('news')
await gotoAndWait('https://example.com', { timeout: 30 })

const snapshot = await snapshotText()
const titles = await js(String.raw`() => {
  const items = [...document.querySelectorAll('h2')]
  return items.map(el => el.innerText)
}`)

cliLog('Found titles:', titles)
await click('@1')                 // click first ref from snapshot
await snapshotText()              // re‑snapshot after navigation

```

Per the SKILL.md documentation, each `snapshotText()` call **rebuilds the ref map**, ensuring `@N` references stay fresh across navigation or dynamic updates. When a ref expires, helpers fall back to stable locators automatically [[SKILL.md#L71‑L84]](https://github.com/citrolabs/ego-lite/blob/main/skills/ego-browser/SKILL.md#L71).

## Visual Workflow: Coordinate‑Based Control for Canvas Apps

The **visual workflow** targets pages where the DOM lies—canvas‑heavy editors, virtualized spreadsheets, maps, and design tools where the true editing surface lives in pixels, not markup.

### When to Use Visual Mode

| Scenario | Why DOM fails | Visual solution |
|----------|-------------|---------------|
| Google Docs/Sheets | Content rendered to `<canvas>` or off‑screen buffers | Screenshot‑based coordinate interaction |
| Figma, Miro, Excalidraw | Vector graphics, no semantic element tree | Pixel‑precise click, drag, keyboard |
| Map interfaces (Google Maps, Mapbox) | Tiles as images, markers overlaid | Viewport coordinate navigation |
| Virtualized lists (Notion, Asana) | DOM nodes recycled, positions unstable | Screenshot verification + coordinate actions |

### Visual Workflow Helpers

From [`src/helpers.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/helpers.ts), visual primitives include:

- `captureScreenshot()` – viewport image capture
- `click([x, y])` – coordinate‑based clicking
- `doubleClick([x, y])`, `moveMouse([x, y])` – gesture simulation
- `pressKey()`, `typeText()` – keyboard injection

### Visual Workflow Example

```javascript
await gotoAndWait('https://canvas-app.example.com')
await captureScreenshot()

await click([200, 150])           // start point
await moveMouse([400, 300])      // drag to end point

await captureScreenshot()         // verify the drawing

```

Because the DOM is unreliable, **screenshots become the source of truth**. Agents validate actions through visual feedback or export/read‑back checks rather than DOM assertions.

## Direct DOM / CDP Workflow: Protocol‑Level Control

The **direct DOM / CDP workflow** exposes the raw browser for scenarios beyond what semantic or visual helpers cover.

### Capabilities of Direct Mode

| Use case | Helper | Underlying mechanism |
|----------|--------|----------------------|
| Custom DOM traversal, state manipulation | `js(expression)` | `Runtime.evaluate` CDP command |
| Protocol features (cookies, network, security) | `cdp(method, params)` | Raw Chrome DevTools Protocol |
| Bespoke data extraction | `js()` with return values | Full JavaScript execution context |
| Performance profiling, request interception | `cdp('Network.*')`, etc. | Direct CDP domain access |

The `js()` and `cdp()` helpers are implemented in [`src/cdp-eval.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/cdp-eval.ts), wrapping `Runtime.evaluate` and generic protocol methods. The **Browser Runtime** ([`src/browser-runtime.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/browser-runtime.ts)) manages CDP transport and session lifecycle, providing the conduit for these low‑level calls.

### Direct DOM/CDP Example

```javascript
// Retrieve all cookies via CDP
const cookies = await cdp('Network.getAllCookies')
cliLog('Cookies:', cookies)

// Custom DOM extraction not covered by snapshotText
const shadowData = await js(String.raw`() => {
  const host = document.querySelector('my-component')
  return host?.shadowRoot?.innerHTML
}`)

```

## Combining Workflows in Practice

Real automation tasks mix workflows strategically. A typical pattern:

1. **Observe** – semantic snapshot or screenshot
2. **Act** – appropriate workflow helper
3. **Verify** – follow‑up snapshot, screenshot, or DOM query
4. **Report** – `cliLog(...)` for observability

### Mixed Workflow Example

```javascript
await gotoAndWait('https://form.example.com')

// Semantic: fill form using accessible refs
await snapshotText()
await fillInput('@12', 'John Doe')
await click('@15')                // submit button

// Visual: confirm success state appearance
await captureScreenshot()

```

This flexibility—**semantic for structure, visual for rendering, direct DOM/CDP for edge cases**—lets agents adapt to any page architecture without abandoning the ego‑browser harness.

## Key Source Files

| File | Responsibility |
|------|--------------|
| [`src/helpers.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/helpers.ts) | Public API surface (`snapshotText`, `click`, `js`, `captureScreenshot`, etc.) |
| [`src/element-resolver.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/element-resolver.ts) | Ref/locator/coordinate → CDP element ID resolution |
| [`src/browser-runtime.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/browser-runtime.ts) | CDP connection, sessions, event buffering |
| [`src/cdp-eval.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/cdp-eval.ts) | `js()` and `cdp()` helper implementations |
| [`skills/ego-browser/SKILL.md`](https://github.com/citrolabs/ego-lite/blob/main/skills/ego-browser/SKILL.md) | Canonical workflow documentation and usage patterns |

## Summary

- **Semantic workflow** uses `snapshotText()` and `@N` refs for compact, DOM‑based automation of standard websites.
- **Visual workflow** uses `captureScreenshot()` and `[x, y]` coordinates for canvas apps and virtualized interfaces where DOM structure is unreliable.
- **Direct DOM/CDP workflow** uses `js()` and `cdp()` for raw JavaScript execution and protocol‑level operations.
- Workflows combine freely in single tasks; the runtime in [`src/browser-runtime.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/browser-runtime.ts) ensures consistent CDP session management.
- The Element Resolver in [`src/element-resolver.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/element-resolver.ts) abstracts away the difference between ref‑based, locator‑based, and coordinate‑based targeting.

## Frequently Asked Questions

### What happens when a semantic ref like `@5` no longer exists?

The helper falls back to the stable `loc=` selector associated with that element. If both fail, the Element Resolver classifies the error as transient (retry after snapshot refresh) or permanent (element genuinely gone). The ref map rebuilds on every `snapshotText()` call, so stale refs resolve automatically [[SKILL.md#L71‑L84]](https://github.com/citrolabs/ego-lite/blob/main/skills/ego-browser/SKILL.md#L71).

### Can I use the visual workflow on any page, or only canvas apps?

You can use coordinate‑based actions anywhere, but they are **brittle** on standard DOM pages where responsive layouts shift element positions. The semantic workflow is preferred for structured pages because refs and locators survive layout changes. Reserve visual mode for sites where DOM traversal is impossible or meaningless.

### How does `js()` differ from running code in `snapshotText()`?

`snapshotText()` runs internal scripts to build the accessibility tree and ref map—you don't control this execution. `js()` is **agent‑controlled** JavaScript via `Runtime.evaluate`, able to return arbitrary data, manipulate state, or access browser APIs intentionally excluded from the semantic abstraction. Both route through [`src/cdp-eval.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/cdp-eval.ts), but `js()` exposes full execution context to the agent.