# How ego-browser's Element Resolver Integrates with the Accessibility Tree for Resilient AI Automation

> Explore ego-browser's element resolver integration with the accessibility tree for resilient AI automation. Learn how this two-tier architecture ensures robust element location against DOM changes.

- Repository: [CitroLabs/ego-lite](https://github.com/citrolabs/ego-lite)
- Tags: architecture
- Published: 2026-07-24

---

**ego-browser implements a two-tier element resolution architecture that first attempts snapshot-based numeric references, then falls back to the Chromium Accessibility (AX) tree via `Accessibility.queryAXTree` to locate elements by semantic role and accessible name, making automation robust against DOM mutations.**

The element resolver in citrolabs/ego-lite serves as the critical bridge between high-level AI agent commands and low-level browser interactions. By integrating tightly with the browser's accessibility tree, the system can locate interactive elements using semantic identifiers like `role=button/name=Submit` rather than brittle CSS selectors or coordinates.

## The Layered Resolution Pipeline

The resolver operates through a layered priority system defined in [`src/element-resolver.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/element-resolver.ts). It first consults the **ref map** ([`src/ref-map.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/ref-map.ts)), which stores snapshot-generated `backendNodeId` values and metadata (role, name, nth) for each `@N` reference, while [`src/ref-state.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/ref-state.ts) manages the mutable snapshot state refreshed during navigation. When these cached IDs become stale or unavailable, the system automatically falls back to **Accessibility tree queries** to recompute the node location.

This dual approach ensures that temporary DOM mutations do not break element references, while the AX tree provides a semantic layer that remains stable even when underlying HTML structures change.

## Parsing Role-Based Locators

When the resolver receives a selector string, the `parseLocator` function (lines 14-18 in [`src/element-resolver.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/element-resolver.ts)) analyzes the format. If the text matches a **role locator** pattern (e.g., `role=button/name=Submit/nth=2`), the parser creates a structured object whose `kind` property is set to `"role"`.

This parsing step differentiates between raw CSS selectors, numeric refs like `@21`, and semantic role queries, ensuring the appropriate resolution path is selected before any CDP commands are issued from [`src/cdp-eval.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/cdp-eval.ts).

## Querying the AX Tree via CDP

For role locators, the resolver invokes `queryRoleLocatorBackendNodeIds`, which delegates to `findBackendNodeIdsByRoleName`. This function issues the Chrome DevTools Protocol (CDP) command **`Accessibility.queryAXTree`** via the internal `send` helper (defined in [`src/cdp-eval.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/cdp-eval.ts)) to request all backend-node IDs matching the specified role, accessible name, and optional `nth` index (lines 22-38 in [`src/element-resolver.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/element-resolver.ts)).

The returned list maintains the AX tree's natural document order. If the locator specifies an `nth` parameter, the resolver selects the precise index from this ordered list; otherwise, it defaults to the last matching element.

## From Backend Nodes to Screen Geometry

Once a valid `backendNodeId` is obtained—either from the ref cache or the AX tree query—the resolver calls `resolveElementCenter`. This function invokes **`DOM.getBoxModel`** to retrieve the element's geometry (lines 70-99 in [`src/element-resolver.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/element-resolver.ts)).

If the node lacks a box model (indicating the element is hidden, not yet rendered, or detached), the resolver throws an `ElementResolutionError` marked as **transient**. This classification allows higher-level automation loops to implement intelligent retry logic rather than failing permanently.

## Handling Stale References and Dynamic DOMs

When a cached ref's `backendNodeId` throws an error or returns `undefined`—common during single-page application transitions—the resolver enters its fallback mode (lines 100-119 in [`src/element-resolver.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/element-resolver.ts)). It discards the stale ID and recomputes the node by re-executing `findBackendNodeIdByRoleName` against the live AX tree.

After obtaining a fresh backend-node ID, the resolver repeats the `DOM.getBoxModel` call to ensure the coordinates reflect the current DOM state. This mechanism decouples automation scripts from timing issues related to page loading or JavaScript hydration.

## Resolving Remote Object IDs

Certain operations require a remote object handle rather than just coordinates. For these cases, the resolver prefers cached backend-node IDs but falls back to the AX-tree lookup when necessary. It then issues **`DOM.resolveNode`** to obtain an `objectId` that can be passed to subsequent CDP methods like `DOM.scrollIntoView` or `Runtime.callFunctionOn` (lines 150-190 in [`src/element-resolver.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/element-resolver.ts)).

This resolution path ensures that even when starting from a stale `@N` reference, the system can acquire a fresh handle to the semantically equivalent element without requiring the AI agent to regenerate the selector.

## Error Handling and Retry Semantics

The resolver wraps all selector evaluation errors in `ElementResolutionError` instances, explicitly categorizing them as **transient** or **permanent** (lines 41-61 in [`src/element-resolver.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/element-resolver.ts)). Transient errors—typically arising from uninitialized box models or temporary DOM detachment—signal to calling functions in [`src/driver/locator.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/driver/locator.ts) that a retry is appropriate. Permanent errors indicate invalid selectors or definitively missing elements, triggering immediate failure.

This distinction enables robust wait-and-retry loops for asynchronous page behavior while preventing infinite loops on genuinely broken selectors.

## Practical Implementation Examples

The following examples demonstrate the resolver API in action:

```javascript
// Resolve the center coordinates of a semantic button
const center = await ego.resolveElementCenter(
  "role=button/name=Submit/nth=2"
);
// Returns: { x: 432, y: 278, sessionId: "1234.1" }

```

```javascript
// Resolve an objectId for a ref that may have become stale
const { objectId, sessionId } = await ego.resolveElementObjectId("@21");
// The objectId can now be used with CDP methods like DOM.scrollIntoView

```

## Summary

- **ego-browser** uses a dual-resolution strategy combining snapshot refs ([`src/ref-map.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/ref-map.ts)) with live **Accessibility tree** queries.
- The resolver parses locators like `role=button/name=Submit` in [`src/element-resolver.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/element-resolver.ts) and issues **`Accessibility.queryAXTree`** to find matching backend-node IDs.
- Geometry extraction relies on **`DOM.getBoxModel`**, while stale references trigger automatic AX-tree recomputation via `findBackendNodeIdsByRoleName`.
- **Transient errors** (hidden elements, loading states) allow retry logic, while **permanent errors** fail fast.
- The architecture shields AI agents from DOM volatility by relying on semantic accessibility properties rather than structural selectors.

## Frequently Asked Questions

### What is the difference between a ref and a role locator in ego-browser?

A **ref** (e.g., `@21`) is a numeric snapshot reference stored in [`src/ref-map.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/ref-map.ts) that maps to a cached `backendNodeId` from a previous page state. A **role locator** (e.g., `role=button/name=Submit/nth=2`) is a semantic query resolved at runtime via **`Accessibility.queryAXTree`**. The resolver prefers refs for performance but falls back to role locators when refs become stale or when the selector explicitly uses role syntax.

### How does ego-browser handle elements that are not yet rendered?

When `DOM.getBoxModel` returns no geometry for an element, the resolver throws an `ElementResolutionError` with the **transient** classification. This signals to the calling automation layer in [`src/driver/locator.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/driver/locator.ts) that the element exists in the accessibility tree but is not yet visually rendered, prompting a retry after a brief interval rather than immediate failure.

### Why does the resolver use the accessibility tree instead of CSS selectors?

The **accessibility tree** reflects the semantic structure of the page (roles, accessible names, ARIA properties) rather than implementation details like CSS classes or DOM IDs. As implemented in [`src/element-resolver.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/element-resolver.ts), this approach makes automation resilient to dynamic DOM mutations, framework re-renders, and obfuscated class names common in modern web applications.

### What CDP commands are essential to the element resolution process?

The resolver relies on three primary Chrome DevTools Protocol commands: **`Accessibility.queryAXTree`** for semantic node discovery, **`DOM.getBoxModel`** for coordinate extraction, and **`DOM.resolveNode`** for obtaining remote object handles. These are orchestrated through the internal `send` helper in [`src/cdp-eval.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/cdp-eval.ts).