Semantic vs Visual vs Direct DOM Workflows in ego-browser: A Complete Guide

ego-browser provides three distinct interaction models—semantic (accessibility tree), visual (pixel coordinates), and direct DOM (raw JavaScript/CDP)—allowing agents to choose the most reliable method for observing and manipulating web pages based on the site’s architecture.

The ego-browser package from the citrolabs/ego-lite repository offers flexible automation strategies for web agents. Understanding the differences between semantic, visual, and direct DOM workflows is essential for building robust browser automation that handles everything from standard forms to complex canvas-based applications. Each workflow targets specific observation layers and state management patterns, as implemented in src/helpers.ts and src/browser-runtime.ts.

The Three ego-browser Interaction Models

Semantic Workflow: Accessibility Tree Navigation

The semantic workflow captures a structured representation of the page using the browser’s accessibility tree. Calling await snapshotText() returns a JSON representation of accessibility/DOM nodes annotated with numeric references ([ref=N]) and location strings (loc=…).

This approach works best on ordinary sites that expose real DOM controls like forms, links, tables, and lists. According to the source code in src/element-resolver.ts, the runtime builds a "ref map" from the accessibility tree, allowing actions like click('@N') or fillInput('@N', …) to target specific elements reliably.

Key characteristics include:

• Observation layer: Uses the browser’s accessibility (AX) tree
• Targeting: References (@N) or locator strings (loc=label:Search)
• Core helpers: snapshotText(), click(), fillInput(), pageInfo()

Visual Workflow: Pixel-Level Coordinate Interaction

The visual workflow operates on bitmap data rather than DOM structure. By calling await captureScreenshot(), agents capture a pixel-level image of the viewport and drive interactions using absolute coordinates and keyboard events.

This method is essential for canvas-like editors, rich-text tools, video conferencing interfaces, maps, and whiteboards—any application where the semantic tree is incomplete or virtualized (e.g., Google Docs, Figma, or Notion). As implemented in src/browser-runtime.ts, this workflow manages screenshot buffering and event queues independently of the DOM.

Key characteristics include:

• Observation layer: Bitmap screenshot stored in memory
• Targeting: Pixel coordinates [x, y]
• Core helpers: captureScreenshot(), click([x, y]), doubleClick([x, y]), pressKey(), typeText()

Direct DOM Workflow: Raw JavaScript and CDP Execution

The direct DOM workflow bypasses both semantic and visual abstraction layers to execute raw JavaScript or Chrome DevTools Protocol (CDP) commands directly inside the page context. Use await js(…) for arbitrary script execution or await cdp(…) for low-level protocol features.

This approach excels when you need compact data extraction, custom DOM traversal, or access to browser features not exposed by higher-level helpers. The implementation in src/cdp-eval.ts wraps user code in an IIFE and transports results back to the Node side.

Key characteristics include:

• Observation layer: Direct page context execution
• Targeting: CSS selectors, XPath, or custom logic inside js()
• Core helpers: js(), cdp()

Critical Architectural Differences

State Stability and Invalidation

Each workflow maintains state differently, impacting how you structure automation loops:

• Semantic refs (@N) are only valid for the most recent snapshotText() call. Any DOM mutation invalidates reference numbers, requiring a fresh snapshot after actions that change the page structure.
• Visual coordinates remain valid as long as the viewport size and page layout remain constant. Navigation or window resizing requires recapturing the screenshot.
• Direct DOM code executes immediately without persistent refs. Variables persist across calls only if you manually return and store data on the Node side.

Error Handling Patterns

Error types vary by workflow, as defined in src/element-resolver.ts and related modules:

• Semantic: Raises ElementResolutionError with a transient flag when references are missing, signaling the need for a re-snapshot.
• Visual: Throws coordinate-out-of-bounds errors when click positions exceed screenshot dimensions.
• Direct DOM: Surfaces JavaScript exceptions from inside the js() IIFE directly to the caller.

Performance Characteristics

The semantic workflow requires parsing the accessibility tree into a ref map, adding overhead for large pages. The visual workflow incurs screenshot encoding/decoding costs. The direct DOM workflow offers the lowest latency for data extraction but requires manual element resolution.

Practical Implementation Examples

Semantic Workflow: Form Interaction

// Observe the page structure
const tree = await snapshotText();

// Target by accessibility label using loc= syntax
await fillInput('loc=label:Search', 'ego-browser');

// Click dynamically generated ref after DOM update
await click('loc=text:ego-browser');

This example uses snapshotText() and locator strings (loc=) from src/helpers.ts to interact with standard form elements.

Visual Workflow: Canvas Drawing

// Capture reference image
const img = await captureScreenshot();

// Interact using measured pixel coordinates
await click([120, 45]);        // Select pen tool
await click([300, 200]);       // Mouse down
await moveMouse([500, 200]);   // Drag
await mouseUp();               // Release

Relies on captureScreenshot() and coordinate-based clicking for applications with virtualized rendering surfaces.

Direct DOM Workflow: Data Extraction

const data = await js(String.raw`(() => {
  const rows = [...document.querySelectorAll('table tr')];
  return rows.map(r => ({
    cells: [...r.cells].map(c => c.innerText.trim())
  }));
})()`);

cliLog('Table data', data);

Executes arbitrary DOM queries via js() as implemented in src/cdp-eval.ts, returning structured JSON to the automation context.

Recommended Workflow Composition

According to skills/ego-browser/SKILL.md, realistic automation scripts often combine all three approaches. Start with a semantic snapshot to locate navigation elements, switch to visual coordinates for canvas interactions, and finish with a js() probe to verify results. The architecture encourages agents to pick the most reliable method first and fall back to lower-level workflows only when necessary.

Summary

• Semantic workflow uses snapshotText() and accessibility tree refs (@N) for reliable interaction with standard DOM elements, raising ElementResolutionError when refs become stale.
• Visual workflow leverages captureScreenshot() and pixel coordinates [x, y] to automate canvas-based applications where the accessibility tree is incomplete or virtualized.
• Direct DOM workflow provides maximum flexibility through js() and cdp() helpers, executing raw JavaScript for custom data extraction and low-level browser control.
• State invalidation differs fundamentally: semantic refs expire on DOM changes, visual coordinates expire on resize/navigation, and direct DOM requires manual state management.
• The src/element-resolver.ts, src/browser-runtime.ts, and src/cdp-eval.ts files implement the core logic for each respective workflow.

Frequently Asked Questions

What is the semantic workflow in ego-browser?

The semantic workflow is a high-level interaction model that captures the browser’s accessibility tree via snapshotText(), annotating interactive elements with numeric references ([ref=N]). It allows agents to click, fill, and navigate using these stable references or locator strings (loc=label:…), making it ideal for standard web forms and document structures. When references become invalid due to DOM mutations, the system raises ElementResolutionError to prompt a fresh snapshot.

When should I use the visual workflow instead of semantic?

Use the visual workflow when automating applications with incomplete or virtualized accessibility trees, such as Google Docs, Figma, whiteboards, or map interfaces. Instead of parsing DOM nodes, this workflow calls captureScreenshot() and drives interactions via pixel coordinates [x, y]. This approach is necessary when the page renders content to a canvas element or uses complex JavaScript-based positioning that obscures the underlying DOM structure from the accessibility tree.

How does error handling differ between the three workflows?

Each workflow surfaces distinct error types based on its observation layer. The semantic workflow raises ElementResolutionError with a transient property when reference numbers cannot be resolved, indicating the need to re-snapshot. The visual workflow throws bounds-checking errors when coordinates exceed screenshot dimensions. The direct DOM workflow propagates raw JavaScript exceptions from within the js() IIFE, requiring standard try/catch blocks to handle DOM query failures or runtime errors.

Can I combine different workflows in a single automation script?

Yes, combining workflows is the recommended pattern for complex automation tasks. According to the official documentation in skills/ego-browser/SKILL.md, agents should start with the semantic workflow for navigation and form filling, switch to visual coordinates for canvas or drawing operations, and use js() or cdp() for data validation or extraction that requires custom DOM traversal. This hybrid approach maximizes reliability while handling edge cases that single-workflow strategies cannot address.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}