Interacting with Canvas-Like Applications in ego-browser: Viewport Coordinate Workflow

The recommended workflow for interacting with canvas-like applications in ego-browser uses viewport coordinates rather than CSS selectors, leveraging the MouseTarget API in src/driver/pointer.ts to click, drag, hover, and scroll at specific pixel positions.

Canvas-based web applications—including HTML <canvas> elements, interactive maps, and custom drawing surfaces—do not expose individual DOM nodes for their internal interactive regions. In the citrolabs/ego-lite repository, ego-browser solves this challenge through a coordinate-based interaction model that operates directly on viewport positions using the mouse-helper APIs.

Why Canvas Elements Require a Different Approach

Standard CSS selectors can locate the canvas container itself, but they cannot target specific pixels or shapes drawn inside the rendering surface. When automating interactions with map widgets, drawing applications, or game interfaces, you need precision at the pixel level. The ego-browser driver circumvents the DOM limitation by accepting viewport coordinates as first-class interaction targets, treating the canvas as a spatial grid rather than a tree of elements.

The Viewport Coordinate Workflow

The workflow centers on the MouseTarget type defined in src/driver/pointer.ts, which resolveMouseTarget processes to determine exact screen positions. You can specify targets in three distinct forms depending on your precision requirements.

Selector-Based Targeting

Pass a CSS selector or @ref string to resolve to the element’s center. This works for clicking the canvas element itself when you don’t need sub-element precision.

await click("canvas#map", { label: "Activate canvas" });

Absolute Coordinate Targeting

Provide an explicit [x, y] array or {x, y} object for direct viewport coordinates. This bypasses element resolution entirely, sending the mouse event to specific screen pixels regardless of what element sits there.

await click({ x: 500, y: 400 }, { label: "Click absolute position" });

Relative Offset Targeting

Combine a selector with an offset object to calculate coordinates relative to an element’s top-left corner. This is the preferred method for canvas interactions, as it anchors your coordinates to the canvas position while allowing precise internal targeting.

The elementTopLeft resolution (handled internally by src/element-resolver.ts) locates the canvas origin, then adds your specified {x, y} offset to determine the final interaction point.

Implementing Canvas Interactions

Follow this three-step process to interact with canvas-like applications reliably:

1. Identify the canvas element using a stable selector (e.g., "canvas#map").
2. Resolve its top-left corner via resolveMouseTarget to establish the coordinate origin.
3. Add the required offset (pixel coordinates inside the canvas) and pass the combined MouseTarget to the appropriate helper function.

Clicking Specific Canvas Locations

Use the selector-plus-offset pattern to click individual tiles, drawing tools, or game entities:

// 1️⃣ Click a point inside a canvas
await click(
  { selector: "canvas#map", x: 150, y: 200 },   // 150 px right, 200 px down from top‑left
  { label: "Select map tile" }
);

Dragging Across Canvas Surfaces

Supply an array of MouseTarget objects to drag for drawing shapes or panning maps:

// 2️⃣ Drag across a canvas (e.g., to draw a shape)
await drag(
  [
    { selector: "canvas#draw", x: 30, y: 30 },
    { selector: "canvas#draw", x: 130, y: 130 },
    { selector: "canvas#draw", x: 230, y: 30 }
  ],
  { label: "Draw triangle" }
);

Hovering for Tooltips

Trigger canvas-specific hover states or tooltips without clicking:

// 3️⃣ Hover over a canvas location (useful for tooltips)
await hover({ selector: "canvas#map", x: 400, y: 300 }, { label: "Show tooltip" });

Scrolling Canvas-Based Views

Use wheel with absolute or relative coordinates to scroll virtualized lists or zoom map widgets:

// 4️⃣ Scroll a canvas‑based map or virtualized list
await wheel(0, -200, { x: 500, y: 400 }); // scroll up 200 px at point (500, 400)

How Mouse Events Are Dispatched

All mouse actions in src/driver/pointer.ts ultimately route through CDP (Input.dispatchMouseEvent) for maximum fidelity. When the target page is backgrounded or CDP delivery fails, the system falls back to synthetic DOM events, ensuring that custom scrollers, map widgets, and drawing surfaces receive events indistinguishable from genuine user input.

The wheel helper specifically respects element visibility and focus states. If CDP injection is unavailable, it synthesizes a WheelEvent on the element under the specified point, maintaining compatibility with canvas-based zoom and pan controls.

Summary

• Canvas applications lack addressable DOM nodes for internal elements, requiring coordinate-based interaction.
• Three targeting modes exist: pure selectors, absolute coordinates, and selector-plus-offset (recommended for canvases).
• Source files powering this workflow include src/driver/pointer.ts for the API, src/element-resolver.ts for coordinate calculation, and src/format.ts (line 477) for documentation.
• Event delivery uses CDP by default with synthetic event fallbacks for backgrounded tabs.
• Practical applications include clicking map tiles, drawing shapes via drag sequences, triggering canvas tooltips, and scrolling virtualized views.

Frequently Asked Questions

How do I click on a specific pixel inside a canvas element?

Pass an object containing both the selector and relative x, y coordinates to any mouse helper. For example, { selector: "canvas#game", x: 100, y: 50 } clicks 100 pixels from the left edge and 50 pixels from the top of the canvas. The driver resolves the selector to get the canvas position, then adds your offset to calculate the absolute viewport coordinates.

Can I drag across multiple points on a canvas using ego-browser?

Yes. The drag helper in src/driver/pointer.ts accepts an array of MouseTarget objects, allowing you to define complex paths. Each point in the array can use different targeting modes, though typically you’ll use the same canvas selector with varying {x, y} offsets to draw shapes or pan across maps.

What happens if the page is backgrounded during canvas interaction?

When the browser page is not focused, CDP (Input.dispatchMouseEvent) may not deliver events reliably. In this case, ego-browser falls back to synthetic DOM events created via JavaScript injection. The wheel helper specifically checks visibility and synthesizes WheelEvent instances on elements under the target coordinates, ensuring canvas scrollers and custom widgets still receive input.

Where are the mouse helper APIs defined in the ego-browser source code?

The primary mouse interaction APIs—including click, drag, hover, and wheel—are defined in src/driver/pointer.ts. The MouseTarget type and resolution logic reside there as well, while src/element-resolver.ts handles the translation of selectors and offsets into concrete DOM rectangles and viewport coordinates.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}