How to Interact with Pages Using Accessibility Tree and Element Refs

Use browse snapshot to capture the accessibility tree and obtain stable element refs like @0-5, then pass those refs to interaction commands such as browse click and browse fill for deterministic browser automation.

Browser automation in the browserbase/skills repository leverages an accessibility-first workflow to eliminate the fragility of CSS selectors. Instead of targeting elements by class or ID, this approach reads the page's accessibility tree to generate stable references that persist across page states. This guide explains how to interact with pages using the accessibility tree and element refs according to the implementation in skills/browser/REFERENCE.md and skills/browser/SKILL.md.

Understanding the Accessibility-First Architecture

The browser automation system runs as a daemon-based CLI that auto-starts on the first command. According to skills/browser/REFERENCE.md, the architecture supports two execution modes that both rely on the accessibility tree for element identification.

The local mode launches a clean Chrome instance directly on your development machine. The remote mode connects to a cloud-hosted Chromium instance on Browserbase when the BROWSERBASE_API_KEY environment variable is set. Both modes expose the same interface, ensuring scripts work identically whether testing locally or running against anti-bot protected production sites.

Local vs. Remote Execution Modes

The daemon handles session management automatically. In local mode, the browser launches headless or headed based on your configuration. In remote mode, the daemon establishes a WebSocket connection to Browserbase infrastructure, allowing you to automate browsers in the cloud while still using the same local CLI commands.

Capturing the Accessibility Tree with browse snapshot

The browse snapshot command extracts the page's accessibility tree and embeds unique element refs for every interactive node. This is the recommended way to understand page state because it is fast, structured, and independent of visual rendering as noted in skills/browser/SKILL.md.

Each ref follows the format @<node-id>-<child-id>, such as @0-5 or @2-12. The command outputs a hierarchical view of the page where buttons, links, and form fields are annotated with their respective refs. This map is cached after each snapshot, creating a stable reference point for subsequent interactions.

Inspecting Cached Element Refs

After running a snapshot, you can inspect the cached ref map without re-running the full extraction using the browse refs command. This is useful when you need to verify available references before executing an interaction or when debugging a multi-step workflow.

Interacting with Elements Using Refs

Interaction commands including click, type, fill, and press accept an element ref directly rather than a CSS selector. This ensures actions target exactly the element that appeared in the last snapshot, eliminating race conditions and DOM mutation issues common in traditional selector-based automation.

Clicking Elements by Ref

To click an element, pass its ref to the browse click command. The following example demonstrates opening a page, capturing refs, and clicking a specific button:

# Load the target page

browse open https://example.com

# Capture tree and generate refs

browse snapshot

# Click the element with ref @0-5

browse click @0-5

# Verify the interaction succeeded

browse snapshot

The browse snapshot output shows the element hierarchy and assigns the ref @0-5 to the target button.

Filling Forms with Accessibility References

Form interactions work identically to clicks. After capturing a snapshot, pass the input element's ref to browse fill or browse type. While some commands technically support traditional CSS selectors, using refs is strongly recommended for reliability. The following example fills credentials and submits:

browse open https://myapp.com/login
browse snapshot
browse fill "#email" "user@example.com"
browse fill "#password" "s3cr3t"
browse press Enter
browse snapshot

Keyboard-Only Navigation

For accessibility-focused workflows or applications where mouse events are insufficient, use browse press with keyboard keys. You can navigate via Tab and activate elements with Enter:

browse open https://example.com
browse snapshot
browse press Tab
browse press Enter

This pattern moves focus to the next focusable element and activates it, mimicking keyboard-only user behavior.

Implementing the Snapshot-First Workflow

A deterministic automation loop follows a strict cycle. First, use browse open <url> to load the page. Second, execute browse snapshot to retrieve the current tree and ref map. Third, choose the appropriate ref and issue an interaction command like browse click @0-5. Fourth, run browse snapshot again to verify the UI change and obtain fresh refs for the next step. Repeat this cycle until the task completes.

This workflow is demonstrated practically in skills/browser/EXAMPLES.md and forms the foundation for higher-level skills like autobrowse and ui-test, both of which depend on the accessibility tree for deterministic assertions.

Handling Stale Refs and Debugging

If an error such as "Element ref not found (e.g., @0-5)" occurs, the page DOM has likely mutated since the last snapshot. According to skills/browser/REFERENCE.md, you must rerun browse snapshot to obtain fresh refs when the page structure changes. This error handling ensures you never interact with outdated elements that may have moved or been removed from the accessibility tree.

Summary

• Element refs (e.g., @0-5) are stable identifiers generated from the accessibility tree via browse snapshot.
• The daemon-based CLI supports both local Chrome instances and remote Browserbase connections using BROWSERBASE_API_KEY.
• Interaction commands (click, fill, press, type) accept refs directly, eliminating selector fragility.
• Always follow the snapshot-first workflow: capture refs, interact, then verify with a new snapshot.
• Stale refs must be refreshed by running browse snapshot again when the page state changes.

Frequently Asked Questions

What format do element refs use in browserbase/skills?

Element refs use the format @<node-id>-<child-id>, such as @0-5 or @2-12. These are generated by the accessibility tree parser and cached after each browse snapshot execution, as documented in skills/browser/REFERENCE.md.

How do I resolve "Element ref not found" errors?

This error indicates the page structure changed since your last snapshot, invalidating cached refs. Run browse snapshot again to generate a fresh ref map. The new output will contain updated references reflecting the current accessibility tree state.

Can I use CSS selectors instead of element refs?

While some commands technically accept CSS selectors (such as browse fill "#email"), the accessibility-first workflow specifically recommends using element refs for all interactions. Refs are tied to the accessibility tree and remain stable regardless of dynamic class changes or DOM restructuring.

What is the difference between local and remote daemon modes?

Local mode launches a Chrome browser instance directly on your machine, suitable for development and testing. Remote mode connects to a Browserbase-hosted Chromium instance when BROWSERBASE_API_KEY is configured, enabling automation of anti-bot protected sites without local browser overhead. Both modes use identical commands and the same accessibility tree extraction logic.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}