How the Ego-Browser Snapshot System Uses Action Marks and Stable Locators

The ego-browser snapshot system generates semantic page representations with temporary numeric refs, persistent stable locators (CSS, XPath, text, ARIA role), and action marks that indicate valid interactions—enabling AI agents to interact reliably across multiple execution rounds.

The ego-browser package in the citrolabs/ego-lite repository provides a Playwright-based automation framework designed specifically for AI agents. At its core lies a sophisticated snapshot system that transforms raw DOM structures into action-ready, durable representations. This article explains how two key features—action marks and stable locators—work together to make web automation deterministic and robust.

---

The Two Layers of Snapshot Enrichment

When you call page.snapshot() or page.snapshotRaw(), the runtime produces a structured object containing three distinct identifier layers. Understanding their differences is essential for building reliable agents.

Temporary Refs: Short-Lived Numeric Identifiers

The Ref-Map (src/ref-map.ts) generates temporary numeric identifiers (@1, @2, @3, etc.) that map directly to backend DOM nodes. These refs are:

• Rebuilt on every snapshot cycle
• Valid only within a single snapshot's lifecycle
• Subject to a ~2-second TTL or immediate invalidation on navigation/DOM mutation

Use refs for quick prototyping, but never persist them across agent steps.

Stable Locators: Persistent, Human-Readable Selectors

The Element-Resolver (src/element-resolver.ts) analyzes each DOM node and constructs the most reliable persistent locator. According to the source in src/format.ts (lines 402–418), these locators survive page refreshes and structural changes.

Available locator strategies include:

Strategy	Format	Best For
CSS selector	loc=css:#submit-btn	Elements with stable IDs or classes
XPath	xpath=//button[text()='Login']	Complex hierarchical relationships
Visible text	text=Submit	Unique, user-visible labels
ARIA role	loc=role:button[name="Login"]	Accessibility-first identification

The validator in src/learning/validate-learning-format.ts (line 235) explicitly warns against "temporary snapshot ref; use stable locators instead"—making this a first-class framework concern.

---

Action Marks: Encoding Valid Interactions

Action marks are metadata tags attached to snapshot entries that indicate what operations an element supports. During snapshot construction, the runtime inspects element properties:

• tabindex ≥ 0 or href present → click action
• <input>, <textarea>, contenteditable → type action
• <select> → select action

When present, the action field contains a structured object like { type: "click" }. This enables two critical optimizations:

1. Direct invocation: Agents can call helpers (click, type, select) without re-resolving element capabilities
2. Validation: The runtime can reject invalid operations before executing CDP commands

---

Snapshot Methods: Raw vs. Semantic

The framework exposes two snapshot methods, both declared in src/format.ts:

page.snapshotRaw(options?) → Promise<object>

Returns the complete structured snapshot with full metadata:

// Get raw snapshot with full control
const raw = await page.snapshotRaw({ maxResultLength: 2000 });

console.log(raw.content);  // HTML markup
console.log(raw.refs);
// [
//   { id: 1, loc: 'css:#loginBtn', action: { type: 'click' } },
//   { id: 2, loc: 'text=Username', action: { type: 'type' } },
//   ...
// ]

page.snapshot(options?) → Promise<string>

Returns a stringified, agent-friendly representation combining refs, stable locators, and action marks:

// Get semantic snapshot as string
const snap = await page.snapshot();
console.log(snap);
// JSON string with both @N refs and loc=... entries

---

Practical Implementation Patterns

Pattern 1: Prefer Stable Locators Over Refs

// ❌ Fragile: temporary ref becomes invalid after any DOM change
await click('@12');

// ✅ Durable: stable locator survives navigation and mutations
await click('css:#login-submit');
await type('loc=role:textbox[name="Email"]', 'user@example.com');

The validate-learning-format.ts validator flags the first pattern with a warning, directing developers toward the second.

Pattern 2: Leverage Action Marks for Efficiency

// Parse snapshot to find actionable elements
const data = JSON.parse(await page.snapshot());

// Filter for clickable buttons with stable locators
const actionable = data.refs.filter(
  r => r.action?.type === 'click' && r.loc?.startsWith('css:')
);

// Execute without re-checking element capabilities
for (const btn of actionable) {
  await click(btn.loc);  // Runtime uses pre-computed action mark
}

Pattern 3: Hybrid Resolution for Complex Flows

import { ElementResolver } from 'ego-browser';

// When a locator needs dynamic refinement
const resolver = new ElementResolver(page);
const refined = await resolver.resolveToStable(
  'xpath=//button[contains(., "Next")]',
  { preferRole: true }
);

// Returns: 'loc=role:button[name="Next step"]'
await click(refined);

---

Key Implementation Files

File	Responsibility
src/format.ts	Declares snapshot() and snapshotRaw() signatures; documents output format
src/driver/observe.ts	Implements snapshot generation and automatic ref refresh
src/element-resolver.ts	Generates stable locators from DOM analysis
src/ref-map.ts	Maintains temporary @N → node mappings
src/ref-state.ts	Tracks snapshot lifecycle and triggers refreshes
src/learning/validate-learning-format.ts	Lints manifests; warns on temporary ref usage

---

Summary

• Temporary refs (@N) provide fast, ephemeral identifiers that break across snapshot cycles—use only for immediate operations.

• Stable locators (loc=css:, xpath=, text=, loc=role:) persist through navigation and DOM changes, forming the backbone of durable automation.

• Action marks encode valid operations directly into snapshots, eliminating redundant capability checks and enabling direct helper invocation.

• The validate-learning-format.ts linter enforces stable locator adoption, catching fragile patterns at development time.

---

Frequently Asked Questions

What happens if I use a temporary ref after it expires?

The runtime throws a resolution error. When ref-state.ts detects expiration (TTL expiry, navigation, or DOM mutation), it clears the ref map. Subsequent attempts to resolve @N identifiers fail, forcing a fresh snapshot. The validator prevents this by warning against temporary refs in learning manifests.

How does the element resolver choose between locator strategies?

It applies a priority cascade: explicit id or data-testid → ARIA role with accessible name → unique CSS class combination → unique text content → XPath structural path. The src/element-resolver.ts implementation scores each candidate by stability likelihood, returning the highest-confidence locator.

Can I customize which action marks get generated?

The snapshot system inspects standard HTML semantics and ARIA attributes to infer actions. Custom elements require explicit ARIA role definitions (role="button", aria-haspopup, etc.) or implementation of the element resolver's extension hooks for non-standard interaction patterns.

Why does snapshotRaw exist alongside `snapshot()?

page.snapshotRaw() returns the complete object graph for programmatic manipulation—useful when building custom agents or debugging. page.snapshot() returns a compact string representation optimized for LLM consumption, with redundant metadata elided. Both draw from the same underlying observation logic in src/driver/observe.ts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}