How Pointer Operations Handle Coordinate Systems in ego-browser: Click, Drag, and Wheel Explained

ego-browser treats all pointer coordinates as CSS-pixel viewport positions, resolving any target format—selector, reference, coordinate pair, or offset object—to an absolute {x, y} point before dispatching to Chrome DevTools Protocol (CDP) or synthetic DOM events.

The page.mouse API in ego-browser provides Playwright-style helpers for automating user interactions. Whether you target an element by selector, pass explicit coordinates, or use a cached reference, the coordinate system remains consistent: CSS pixels measured from the top-left corner of the viewport. This article breaks down how click, drag, and wheel operations interpret and transform coordinates based on the source code in citrolabs/ego-lite.

How Coordinates Are Resolved for Click Operations

The MouseTarget Resolution Pipeline

In src/driver/pointer.ts, the resolveMouseTarget() function (lines 46–53 and 66–78) handles all target-to-coordinate translation. It accepts five MouseTarget formats:

• String selector — 'button.submit'
• Reference handle — @ref returned from previous evaluations
• Coordinate pair — [400, 250] or {x: 400, y: 250}
• Selector with offset — { selector: '#menu', x: 10, y: 5 }

When a selector is provided, the implementation:

1. Waits for the element to appear in the DOM
2. Scrolls it into view if necessary
3. Calls elementCenter() from src/element-resolver.ts to compute viewport coordinates
4. Adds any optional offset to the element's top-left corner (lines 31–35)

The resulting {x, y} is passed to dispatchMouse(), which invokes Input.dispatchMouseEvent through CDP with identical coordinate values (lines 73–84).

// Click the centre of a button
await page.mouse.click('button.submit', { label: 'Submit click' });

// Click at explicit viewport coordinates
await page.mouse.click([400, 250]);

// Click with offset from element's top-left corner
await page.mouse.click({ selector: '#menu', x: 10, y: 5 });

How Drag Operations Build Coordinate Paths

The drag() helper receives an array of MouseTargets, treating each entry as a waypoint in the drag path. According to lines 38–46 in pointer.ts, every waypoint resolves through the same resolveMouseTarget() routine.

The drag execution (lines 57–71) proceeds as follows:

1. Resolve the first target to establish currentMousePoint
2. Dispatch mousePressed via dispatchMouse()
3. For each subsequent target, compute intermediate mouseMoved events
4. Dispatch mouseReleased at the final point

Because each waypoint resolves to a CSS-pixel viewport coordinate before the drag begins, the entire path is pre-calculated. This guarantees consistent behavior even if page layout shifts during the operation.

// Drag from one element to another with optional delay between steps
await page.mouse.drag(
  [
    '#drag-source',                           // start at element centre
    { selector: '#drop-target', x: 20, y: 30 } // offset from target
  ],
  { delay: 50, label: 'Drag item' }
);

How Wheel Operations Handle Coordinates and Fallbacks

CDP vs. Synthetic Event Paths

The wheel() function (lines 86–97 in pointer.ts) accepts deltaX/deltaY scroll offsets plus optional x/y coordinates (defaulting to 0,0). After normalizing values with numberValue() (lines 86–89), it checks isVisibleAndFocused() to determine the dispatch path:

Condition	Implementation	Coordinates Used
Tab is foregrounded	browserCdp() sends Input.dispatchMouseEvent with type mouseWheel	Direct x, y parameters (lines 92–94)
Tab is backgrounded	dispatchSyntheticWheel() evaluates a script in page context	Same x, y passed to document.elementFromPoint(x, y) (lines 35–50)

The synthetic fallback fires a WheelEvent at the element returned by elementFromPoint, maintaining coordinate consistency across both paths.

// Scroll down 400px at current mouse location
await page.mouse.wheel(0, 400);

// Scroll a virtual list at specific viewport coordinates
await page.mouse.wheel(0, 200, { x: 300, y: 600 });

Key Design Guarantees in ego-browser's Coordinate System

Single Resolution, Reuse for Consistency

The implementation resolves each MouseTarget once and stores the resulting point. All subsequent CDP calls or synthetic events use this identical coordinate pair. This design prevents drift from DOM mutations or viewport changes during multi-step operations.

No Device Pixel Confusion

Unlike systems that mix device-independent pixels with device pixels, ego-browser's coordinate system strictly uses CSS pixels. The values you pass or compute match exactly what Input.dispatchMouseEvent receives, eliminating scaling surprises on high-DPI displays.

Offset Behavior on Elements

When using { selector, x?, y? } format, offsets add to the element's top-left corner, not its center. This matches Playwright's behavior and enables precise targeting of sub-element regions.

Source File Reference

File	Purpose
src/driver/pointer.ts	Core pointer helpers: click, dblclick, hover, drag, down, up, wheel
src/element-resolver.ts	elementCenter() and elementTopLeft() calculations
src/cdp-eval.ts	CDP wrapper utilities for dispatchMouse()
src/browser-runtime.ts	Low-level CDP transport via browserCdp()

Summary

• CSS-pixel viewport coordinates are the universal language for all pointer operations in ego-browser
• resolveMouseTarget() in pointer.ts handles selector resolution, scrolling, and center calculation in one pipeline
• Drag paths pre-resolve all waypoints before execution, ensuring coordinate stability
• Wheel events choose CDP or synthetic dispatch based on tab visibility, using identical coordinates for both
• Offset objects add to element top-left corners, not centers, for predictable positioning

Frequently Asked Questions

What coordinate system does ego-browser use for mouse actions?

ego-browser uses CSS pixels relative to the viewport top-left corner. All coordinates—whether from selectors, explicit values, or references—resolve to this system before dispatch. This matches Playwright's behavior and avoids device-pixel scaling issues.

Can I click at an offset from an element's center rather than its top-left?

Not directly with the { selector, x, y } syntax—offsets always add to the top-left corner. To click offset from center, first obtain the element's center coordinates through evaluation, then pass explicit { x, y } coordinates with your desired adjustment.

Why does drag use an array of MouseTargets instead of start/end coordinates?

The array design supports multi-point drags and complex gestures. Each waypoint resolves independently, allowing drags across elements that may not exist simultaneously in the DOM or that require different resolution strategies.

What happens if the tab loses focus during a wheel operation?

If isVisibleAndFocused() returns false (lines 90–97), wheel() falls back to dispatchSyntheticWheel(). This evaluates JavaScript in the page context to find the element at the specified coordinate and fire a synthetic WheelEvent, maintaining operation success without CDP.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}