How the Pointer Driver in ego-browser Handles Click, Hover, Drag, and Wheel Actions

The pointer driver in ego-browser implements a Playwright-style mouse abstraction over Chrome DevTools Protocol (CDP) with robust fallback to synthetic DOM events when native events cannot be dispatched.

The ego-browser pointer driver (package/ego-browser/src/driver/pointer.ts) provides deterministic browser automation by combining fast CDP mouse events with a probe-based fallback system. This ensures interactions succeed even when the target tab is backgrounded or CDP messages are dropped.

Core Architecture

Every pointer action follows a standardized six-step workflow defined in the source code:

1. Resolve the target — Convert selectors, coordinates, or offset objects into viewport Point objects via resolveMouseTarget (L 602‑L 641)
2. Remember position — Store coordinates in currentMousePoint for subsequent down()/up() calls via rememberMousePoint (L 75‑L 77)
3. Visual highlighting — Optionally show highlights and attach labels via maybeHighlight (L 66‑L 73)
4. Dispatch CDP events — Send Input.dispatchMouseEvent messages through browserCdp via dispatchMouse (L 79‑L 95)
5. Fallback probing — Install lightweight DOM probes; synthesize native events if probes don't fire
6. Error handling — Catch CDP timeouts and re-throw only when fallback fails via isInputDispatchTimeout (L 98‑L 100)

Click and Double-Click Actions

click(target, options?)

The click action resolves the target, then dispatches a precise sequence: mouseMoved → mousePressed → mouseReleased. It installs a click probe via installClickProbe and finishClickProbe (L 24‑L 64) to detect whether the page received a native click event. If the probe never fires, the driver synthesizes MouseEvent instances directly in the page context.

Source: click (L 63‑L 95)

dblclick(target, options?)

The double-click action reuses the click implementation with clickCount: 2, ensuring proper event sequencing for double-click handlers.

Source: dblclick (L 98‑L 104)

// Single click with label for agent state
await click('#submit', { label: 'Submit form' });

// Double-click with default options
await dblclick('#logo');

Hover Action

hover(target, options?)

The hover action resolves the target, remembers the coordinates, and dispatches a single mouseMoved event. It uses installHoverProbe and finishHoverProbe (L 56‑L 70) to verify the hover registered. When CDP fails, it falls back to synthetic mousemove and mouseover events.

Source: hover (L 110‑L 130)

// Hover with visual label
await hover('nav a.settings', { label: 'Open settings' });

Drag Action

drag(points, options?)

The drag action accepts an ordered array of MouseTarget objects and resolves each to screen coordinates. The implementation:

• Dispatches mousePressed at the first point
• Sends interpolated mouseMoved events along the path
• Dispatches mouseReleased at the final point

It installs a mouseup probe via installMouseUpProbe and finishDragProbe (L 92‑L 106, L 103‑L 113) to detect native handling. Missing events trigger synthetic mousedown/mousemove/mouseup sequences.

Source: drag (L 132‑L 190)

// Drag a slider from left edge to center
await drag(
  [
    { selector: '#slider', x: 0, y: 10 },   // start offset
    { selector: '#slider', x: 50, y: 10 }   // end offset
  ],
  { label: 'Adjust slider' }
);

Mouse Button State Helpers

down(options?) and up(options?)

These stateful helpers operate on currentMousePoint — the last remembered mouse position. They directly call dispatchMouse with mousePressed and mouseReleased respectively, enabling manual press-and-hold sequences.

Source: down (L 96‑L 104), up (L 107‑L 115)

// Manual drag sequence
await down();                     // press at current position
await click({ x: 400, y: 300 });  // move to new coordinates
await up();                       // release button

Wheel/Scroll Action

wheel(deltaX?, deltaY?, options?)

The wheel action prioritizes CDP's Input.dispatchMouseEvent with type mouseWheel when the page has focus and visibility. For background tabs, it falls back to dispatchSyntheticWheel (L 128‑L 150): a synthetic WheelEvent on the element under (x, y) followed by window.scrollBy to ensure scroll position updates.

Source: wheel (L 64‑L 86)

// Scroll down 500px at specific coordinates
await wheel(0, 500, { x: 100, y: 200 });

Supporting Utilities

Utility	Purpose	Location
resolveMouseTarget	Converts selectors, coordinates, or offset objects to Point	L 602‑L 642
elementCenter / elementTopLeft	Query element geometry via CDP Runtime.evaluate	L 44‑L 54
waitForSelector	Visibility polling before interaction	src/driver/waits.ts
browserCdp	Low-level CDP wrapper with request timeouts	src/browser-runtime.ts

File Structure

The pointer subsystem spans these key files in package/ego-browser/src/:

• driver/pointer.ts — Core implementation with all action methods and probe logic
• driver/waits.ts — Selector waiting utilities (waitForSelector)
• driver/observe.ts — DOM geometry queries (elementCenter)
• cdp-eval.ts — Synthetic event injection via Runtime.evaluate
• browser-runtime.ts — CDP transport layer (browserCdp)

Summary

• The pointer driver combines CDP native events with DOM probes for maximum reliability
• Six-step workflow standardizes all actions: resolve → remember → highlight → dispatch → probe → handle errors
• Click, hover, drag, and wheel each implement specialized probe pairs for fallback detection
• Synthetic event generation ensures page listeners fire when CDP events cannot be delivered
• Stateful position tracking enables down()/up() to operate without re-specifying coordinates

Frequently Asked Questions

What makes ego-browser's pointer driver different from standard Playwright?

The ego-browser pointer driver adds probe-based fallback verification not present in standard Playwright. According to the citrolabs/ego-lite source code, after dispatching CDP events the driver installs lightweight DOM probes that check whether native events actually fired. If probes remain silent, the driver synthesizes DOM events — a mechanism specifically designed for background tabs and unreceptive browser contexts.

How does the driver handle clicking elements in background tabs?

When CDP's Input.dispatchMouseEvent times out or fails in background tabs, the click probe mechanism detects the failure via finishClickProbe. The driver then uses cdp-eval.ts to inject synthetic MouseEvent instances directly into the page's JavaScript context, ensuring event listeners execute regardless of tab visibility state.

Can I perform complex drag operations without specifying every coordinate?

Yes. The drag action accepts an array of MouseTarget objects that can mix selectors with relative offsets. The driver automatically interpolates movement between points and handles the full mousePressed → mouseMoved sequence → mouseReleased lifecycle, including probe verification at each critical transition.

Why does wheel scrolling require two different implementation paths?

The wheel action uses CDP mouseWheel events for foreground tabs because they trigger native scroll physics and bubble correctly. For background tabs, CDP mouse events may be ignored by the renderer, so the driver falls back to synthetic WheelEvent dispatch combined with direct window.scrollBy manipulation to guarantee scroll position changes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}