# Semantic vs Visual vs Direct DOM Workflows in ego-browser: A Complete Guide

> Explore semantic, visual, and direct DOM workflows in ego-browser. Understand each model to reliably interact with web pages using the best integration method.

- Repository: [CitroLabs/ego-lite](https://github.com/citrolabs/ego-lite)
- Tags: deep-dive
- Published: 2026-07-30

---

**ego-browser provides three distinct interaction models—semantic (accessibility tree), visual (pixel coordinates), and direct DOM (raw JavaScript/CDP)—allowing agents to choose the most reliable method for observing and manipulating web pages based on the site’s architecture.**

The `ego-browser` package from the [citrolabs/ego-lite](https://github.com/citrolabs/ego-lite) repository offers flexible automation strategies for web agents. Understanding the differences between **semantic**, **visual**, and **direct DOM workflows** is essential for building robust browser automation that handles everything from standard forms to complex canvas-based applications. Each workflow targets specific observation layers and state management patterns, as implemented in [`src/helpers.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/helpers.ts) and [`src/browser-runtime.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/browser-runtime.ts).

## The Three ego-browser Interaction Models

### Semantic Workflow: Accessibility Tree Navigation

The **semantic workflow** captures a structured representation of the page using the browser’s accessibility tree. Calling `await snapshotText()` returns a JSON representation of **accessibility/DOM nodes** annotated with numeric references (`[ref=N]`) and location strings (`loc=…`).

This approach works best on ordinary sites that expose real DOM controls like forms, links, tables, and lists. According to the source code in [`src/element-resolver.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/element-resolver.ts), the runtime builds a "ref map" from the accessibility tree, allowing actions like `click('@N')` or `fillInput('@N', …)` to target specific elements reliably.

Key characteristics include:

- **Observation layer**: Uses the browser’s accessibility (AX) tree
- **Targeting**: References (`@N`) or locator strings (`loc=label:Search`)
- **Core helpers**: `snapshotText()`, `click()`, `fillInput()`, `pageInfo()`

### Visual Workflow: Pixel-Level Coordinate Interaction

The **visual workflow** operates on bitmap data rather than DOM structure. By calling `await captureScreenshot()`, agents capture a pixel-level image of the viewport and drive interactions using **absolute coordinates** and keyboard events.

This method is essential for canvas-like editors, rich-text tools, video conferencing interfaces, maps, and whiteboards—any application where the semantic tree is incomplete or virtualized (e.g., Google Docs, Figma, or Notion). As implemented in [`src/browser-runtime.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/browser-runtime.ts), this workflow manages screenshot buffering and event queues independently of the DOM.

Key characteristics include:

- **Observation layer**: Bitmap screenshot stored in memory
- **Targeting**: Pixel coordinates `[x, y]`
- **Core helpers**: `captureScreenshot()`, `click([x, y])`, `doubleClick([x, y])`, `pressKey()`, `typeText()`

### Direct DOM Workflow: Raw JavaScript and CDP Execution

The **direct DOM workflow** bypasses both semantic and visual abstraction layers to execute **raw JavaScript** or **Chrome DevTools Protocol (CDP) commands** directly inside the page context. Use `await js(…)` for arbitrary script execution or `await cdp(…)` for low-level protocol features.

This approach excels when you need compact data extraction, custom DOM traversal, or access to browser features not exposed by higher-level helpers. The implementation in [`src/cdp-eval.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/cdp-eval.ts) wraps user code in an IIFE and transports results back to the Node side.

Key characteristics include:

- **Observation layer**: Direct page context execution
- **Targeting**: CSS selectors, XPath, or custom logic inside `js()`
- **Core helpers**: `js()`, `cdp()`

## Critical Architectural Differences

### State Stability and Invalidation

Each workflow maintains state differently, impacting how you structure automation loops:

- **Semantic refs** (`@N`) are only valid for the most recent `snapshotText()` call. Any DOM mutation invalidates reference numbers, requiring a fresh snapshot after actions that change the page structure.
- **Visual coordinates** remain valid as long as the viewport size and page layout remain constant. Navigation or window resizing requires recapturing the screenshot.
- **Direct DOM** code executes immediately without persistent refs. Variables persist across calls only if you manually return and store data on the Node side.

### Error Handling Patterns

Error types vary by workflow, as defined in [`src/element-resolver.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/element-resolver.ts) and related modules:

- **Semantic**: Raises `ElementResolutionError` with a `transient` flag when references are missing, signaling the need for a re-snapshot.
- **Visual**: Throws coordinate-out-of-bounds errors when click positions exceed screenshot dimensions.
- **Direct DOM**: Surfaces JavaScript exceptions from inside the `js()` IIFE directly to the caller.

### Performance Characteristics

The semantic workflow requires parsing the accessibility tree into a ref map, adding overhead for large pages. The visual workflow incurs screenshot encoding/decoding costs. The direct DOM workflow offers the lowest latency for data extraction but requires manual element resolution.

## Practical Implementation Examples

### Semantic Workflow: Form Interaction

```javascript
// Observe the page structure
const tree = await snapshotText();

// Target by accessibility label using loc= syntax
await fillInput('loc=label:Search', 'ego-browser');

// Click dynamically generated ref after DOM update
await click('loc=text:ego-browser');

```

*This example uses `snapshotText()` and locator strings (`loc=`) from [`src/helpers.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/helpers.ts) to interact with standard form elements.*

### Visual Workflow: Canvas Drawing

```javascript
// Capture reference image
const img = await captureScreenshot();

// Interact using measured pixel coordinates
await click([120, 45]);        // Select pen tool
await click([300, 200]);       // Mouse down
await moveMouse([500, 200]);   // Drag
await mouseUp();               // Release

```

*Relies on `captureScreenshot()` and coordinate-based clicking for applications with virtualized rendering surfaces.*

### Direct DOM Workflow: Data Extraction

```javascript
const data = await js(String.raw`(() => {
  const rows = [...document.querySelectorAll('table tr')];
  return rows.map(r => ({
    cells: [...r.cells].map(c => c.innerText.trim())
  }));
})()`);

cliLog('Table data', data);

```

*Executes arbitrary DOM queries via `js()` as implemented in [`src/cdp-eval.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/cdp-eval.ts), returning structured JSON to the automation context.*

## Recommended Workflow Composition

According to [`skills/ego-browser/SKILL.md`](https://github.com/citrolabs/ego-lite/blob/main/skills/ego-browser/SKILL.md), realistic automation scripts often **combine** all three approaches. Start with a semantic snapshot to locate navigation elements, switch to visual coordinates for canvas interactions, and finish with a `js()` probe to verify results. The architecture encourages agents to **pick the most reliable method first** and fall back to lower-level workflows only when necessary.

## Summary

- **Semantic workflow** uses `snapshotText()` and accessibility tree refs (`@N`) for reliable interaction with standard DOM elements, raising `ElementResolutionError` when refs become stale.
- **Visual workflow** leverages `captureScreenshot()` and pixel coordinates `[x, y]` to automate canvas-based applications where the accessibility tree is incomplete or virtualized.
- **Direct DOM workflow** provides maximum flexibility through `js()` and `cdp()` helpers, executing raw JavaScript for custom data extraction and low-level browser control.
- State invalidation differs fundamentally: semantic refs expire on DOM changes, visual coordinates expire on resize/navigation, and direct DOM requires manual state management.
- The [`src/element-resolver.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/element-resolver.ts), [`src/browser-runtime.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/browser-runtime.ts), and [`src/cdp-eval.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/cdp-eval.ts) files implement the core logic for each respective workflow.

## Frequently Asked Questions

### What is the semantic workflow in ego-browser?

The **semantic workflow** is a high-level interaction model that captures the browser’s accessibility tree via `snapshotText()`, annotating interactive elements with numeric references (`[ref=N]`). It allows agents to click, fill, and navigate using these stable references or locator strings (`loc=label:…`), making it ideal for standard web forms and document structures. When references become invalid due to DOM mutations, the system raises `ElementResolutionError` to prompt a fresh snapshot.

### When should I use the visual workflow instead of semantic?

Use the **visual workflow** when automating applications with incomplete or virtualized accessibility trees, such as Google Docs, Figma, whiteboards, or map interfaces. Instead of parsing DOM nodes, this workflow calls `captureScreenshot()` and drives interactions via pixel coordinates `[x, y]`. This approach is necessary when the page renders content to a canvas element or uses complex JavaScript-based positioning that obscures the underlying DOM structure from the accessibility tree.

### How does error handling differ between the three workflows?

Each workflow surfaces distinct error types based on its observation layer. The **semantic workflow** raises `ElementResolutionError` with a `transient` property when reference numbers cannot be resolved, indicating the need to re-snapshot. The **visual workflow** throws bounds-checking errors when coordinates exceed screenshot dimensions. The **direct DOM workflow** propagates raw JavaScript exceptions from within the `js()` IIFE, requiring standard try/catch blocks to handle DOM query failures or runtime errors.

### Can I combine different workflows in a single automation script?

Yes, combining workflows is the recommended pattern for complex automation tasks. According to the official documentation in [`skills/ego-browser/SKILL.md`](https://github.com/citrolabs/ego-lite/blob/main/skills/ego-browser/SKILL.md), agents should start with the semantic workflow for navigation and form filling, switch to visual coordinates for canvas or drawing operations, and use `js()` or `cdp()` for data validation or extraction that requires custom DOM traversal. This hybrid approach maximizes reliability while handling edge cases that single-workflow strategies cannot address.