How ego-browser's Element Resolver Integrates with the Accessibility Tree for Resilient AI Automation

ego-browser implements a two-tier element resolution architecture that first attempts snapshot-based numeric references, then falls back to the Chromium Accessibility (AX) tree via Accessibility.queryAXTree to locate elements by semantic role and accessible name, making automation robust against DOM mutations.

The element resolver in citrolabs/ego-lite serves as the critical bridge between high-level AI agent commands and low-level browser interactions. By integrating tightly with the browser's accessibility tree, the system can locate interactive elements using semantic identifiers like role=button/name=Submit rather than brittle CSS selectors or coordinates.

The Layered Resolution Pipeline

The resolver operates through a layered priority system defined in src/element-resolver.ts. It first consults the ref map (src/ref-map.ts), which stores snapshot-generated backendNodeId values and metadata (role, name, nth) for each @N reference, while src/ref-state.ts manages the mutable snapshot state refreshed during navigation. When these cached IDs become stale or unavailable, the system automatically falls back to Accessibility tree queries to recompute the node location.

This dual approach ensures that temporary DOM mutations do not break element references, while the AX tree provides a semantic layer that remains stable even when underlying HTML structures change.

Parsing Role-Based Locators

When the resolver receives a selector string, the parseLocator function (lines 14-18 in src/element-resolver.ts) analyzes the format. If the text matches a role locator pattern (e.g., role=button/name=Submit/nth=2), the parser creates a structured object whose kind property is set to "role".

This parsing step differentiates between raw CSS selectors, numeric refs like @21, and semantic role queries, ensuring the appropriate resolution path is selected before any CDP commands are issued from src/cdp-eval.ts.

Querying the AX Tree via CDP

For role locators, the resolver invokes queryRoleLocatorBackendNodeIds, which delegates to findBackendNodeIdsByRoleName. This function issues the Chrome DevTools Protocol (CDP) command Accessibility.queryAXTree via the internal send helper (defined in src/cdp-eval.ts) to request all backend-node IDs matching the specified role, accessible name, and optional nth index (lines 22-38 in src/element-resolver.ts).

The returned list maintains the AX tree's natural document order. If the locator specifies an nth parameter, the resolver selects the precise index from this ordered list; otherwise, it defaults to the last matching element.

From Backend Nodes to Screen Geometry

Once a valid backendNodeId is obtained—either from the ref cache or the AX tree query—the resolver calls resolveElementCenter. This function invokes DOM.getBoxModel to retrieve the element's geometry (lines 70-99 in src/element-resolver.ts).

If the node lacks a box model (indicating the element is hidden, not yet rendered, or detached), the resolver throws an ElementResolutionError marked as transient. This classification allows higher-level automation loops to implement intelligent retry logic rather than failing permanently.

Handling Stale References and Dynamic DOMs

When a cached ref's backendNodeId throws an error or returns undefined—common during single-page application transitions—the resolver enters its fallback mode (lines 100-119 in src/element-resolver.ts). It discards the stale ID and recomputes the node by re-executing findBackendNodeIdByRoleName against the live AX tree.

After obtaining a fresh backend-node ID, the resolver repeats the DOM.getBoxModel call to ensure the coordinates reflect the current DOM state. This mechanism decouples automation scripts from timing issues related to page loading or JavaScript hydration.

Resolving Remote Object IDs

Certain operations require a remote object handle rather than just coordinates. For these cases, the resolver prefers cached backend-node IDs but falls back to the AX-tree lookup when necessary. It then issues DOM.resolveNode to obtain an objectId that can be passed to subsequent CDP methods like DOM.scrollIntoView or Runtime.callFunctionOn (lines 150-190 in src/element-resolver.ts).

This resolution path ensures that even when starting from a stale @N reference, the system can acquire a fresh handle to the semantically equivalent element without requiring the AI agent to regenerate the selector.

Error Handling and Retry Semantics

The resolver wraps all selector evaluation errors in ElementResolutionError instances, explicitly categorizing them as transient or permanent (lines 41-61 in src/element-resolver.ts). Transient errors—typically arising from uninitialized box models or temporary DOM detachment—signal to calling functions in src/driver/locator.ts that a retry is appropriate. Permanent errors indicate invalid selectors or definitively missing elements, triggering immediate failure.

This distinction enables robust wait-and-retry loops for asynchronous page behavior while preventing infinite loops on genuinely broken selectors.

Practical Implementation Examples

The following examples demonstrate the resolver API in action:

// Resolve the center coordinates of a semantic button
const center = await ego.resolveElementCenter(
  "role=button/name=Submit/nth=2"
);
// Returns: { x: 432, y: 278, sessionId: "1234.1" }

// Resolve an objectId for a ref that may have become stale
const { objectId, sessionId } = await ego.resolveElementObjectId("@21");
// The objectId can now be used with CDP methods like DOM.scrollIntoView

Summary

• ego-browser uses a dual-resolution strategy combining snapshot refs (src/ref-map.ts) with live Accessibility tree queries.
• The resolver parses locators like role=button/name=Submit in src/element-resolver.ts and issues Accessibility.queryAXTree to find matching backend-node IDs.
• Geometry extraction relies on DOM.getBoxModel, while stale references trigger automatic AX-tree recomputation via findBackendNodeIdsByRoleName.
• Transient errors (hidden elements, loading states) allow retry logic, while permanent errors fail fast.
• The architecture shields AI agents from DOM volatility by relying on semantic accessibility properties rather than structural selectors.

Frequently Asked Questions

What is the difference between a ref and a role locator in ego-browser?

A ref (e.g., @21) is a numeric snapshot reference stored in src/ref-map.ts that maps to a cached backendNodeId from a previous page state. A role locator (e.g., role=button/name=Submit/nth=2) is a semantic query resolved at runtime via Accessibility.queryAXTree. The resolver prefers refs for performance but falls back to role locators when refs become stale or when the selector explicitly uses role syntax.

How does ego-browser handle elements that are not yet rendered?

When DOM.getBoxModel returns no geometry for an element, the resolver throws an ElementResolutionError with the transient classification. This signals to the calling automation layer in src/driver/locator.ts that the element exists in the accessibility tree but is not yet visually rendered, prompting a retry after a brief interval rather than immediate failure.

Why does the resolver use the accessibility tree instead of CSS selectors?

The accessibility tree reflects the semantic structure of the page (roles, accessible names, ARIA properties) rather than implementation details like CSS classes or DOM IDs. As implemented in src/element-resolver.ts, this approach makes automation resilient to dynamic DOM mutations, framework re-renders, and obfuscated class names common in modern web applications.

What CDP commands are essential to the element resolution process?

The resolver relies on three primary Chrome DevTools Protocol commands: Accessibility.queryAXTree for semantic node discovery, DOM.getBoxModel for coordinate extraction, and DOM.resolveNode for obtaining remote object handles. These are orchestrated through the internal send helper in src/cdp-eval.ts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}