# Interacting with Canvas-Like Applications in ego-browser: Viewport Coordinate Workflow

> Learn the recommended workflow for interacting with canvas-like applications in ego-browser. Use viewport coordinates and the MouseTarget API for efficient clicking, dragging, and scrolling.

- Repository: [CitroLabs/ego-lite](https://github.com/citrolabs/ego-lite)
- Tags: how-to-guide
- Published: 2026-07-30

---

**The recommended workflow for interacting with canvas-like applications in ego-browser uses viewport coordinates rather than CSS selectors, leveraging the MouseTarget API in [`src/driver/pointer.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/driver/pointer.ts) to click, drag, hover, and scroll at specific pixel positions.**

Canvas-based web applications—including HTML `<canvas>` elements, interactive maps, and custom drawing surfaces—do not expose individual DOM nodes for their internal interactive regions. In the `citrolabs/ego-lite` repository, **ego-browser** solves this challenge through a coordinate-based interaction model that operates directly on viewport positions using the mouse-helper APIs.

## Why Canvas Elements Require a Different Approach

Standard CSS selectors can locate the canvas container itself, but they cannot target specific pixels or shapes drawn inside the rendering surface. When automating interactions with map widgets, drawing applications, or game interfaces, you need precision at the pixel level. The ego-browser driver circumvents the DOM limitation by accepting **viewport coordinates** as first-class interaction targets, treating the canvas as a spatial grid rather than a tree of elements.

## The Viewport Coordinate Workflow

The workflow centers on the `MouseTarget` type defined in [`src/driver/pointer.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/driver/pointer.ts), which `resolveMouseTarget` processes to determine exact screen positions. You can specify targets in three distinct forms depending on your precision requirements.

### Selector-Based Targeting

Pass a CSS selector or `@ref` string to resolve to the element’s center. This works for clicking the canvas element itself when you don’t need sub-element precision.

```typescript
await click("canvas#map", { label: "Activate canvas" });

```

### Absolute Coordinate Targeting

Provide an explicit `[x, y]` array or `{x, y}` object for direct viewport coordinates. This bypasses element resolution entirely, sending the mouse event to specific screen pixels regardless of what element sits there.

```typescript
await click({ x: 500, y: 400 }, { label: "Click absolute position" });

```

### Relative Offset Targeting

Combine a selector with an offset object to calculate coordinates relative to an element’s top-left corner. This is the preferred method for canvas interactions, as it anchors your coordinates to the canvas position while allowing precise internal targeting.

The `elementTopLeft` resolution (handled internally by [`src/element-resolver.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/element-resolver.ts)) locates the canvas origin, then adds your specified `{x, y}` offset to determine the final interaction point.

## Implementing Canvas Interactions

Follow this three-step process to interact with canvas-like applications reliably:

1. **Identify the canvas element** using a stable selector (e.g., `"canvas#map"`).
2. **Resolve its top-left corner** via `resolveMouseTarget` to establish the coordinate origin.
3. **Add the required offset** (pixel coordinates inside the canvas) and pass the combined `MouseTarget` to the appropriate helper function.

### Clicking Specific Canvas Locations

Use the selector-plus-offset pattern to click individual tiles, drawing tools, or game entities:

```typescript
// 1️⃣ Click a point inside a canvas
await click(
  { selector: "canvas#map", x: 150, y: 200 },   // 150 px right, 200 px down from top‑left
  { label: "Select map tile" }
);

```

### Dragging Across Canvas Surfaces

Supply an array of `MouseTarget` objects to `drag` for drawing shapes or panning maps:

```typescript
// 2️⃣ Drag across a canvas (e.g., to draw a shape)
await drag(
  [
    { selector: "canvas#draw", x: 30, y: 30 },
    { selector: "canvas#draw", x: 130, y: 130 },
    { selector: "canvas#draw", x: 230, y: 30 }
  ],
  { label: "Draw triangle" }
);

```

### Hovering for Tooltips

Trigger canvas-specific hover states or tooltips without clicking:

```typescript
// 3️⃣ Hover over a canvas location (useful for tooltips)
await hover({ selector: "canvas#map", x: 400, y: 300 }, { label: "Show tooltip" });

```

### Scrolling Canvas-Based Views

Use `wheel` with absolute or relative coordinates to scroll virtualized lists or zoom map widgets:

```typescript
// 4️⃣ Scroll a canvas‑based map or virtualized list
await wheel(0, -200, { x: 500, y: 400 }); // scroll up 200 px at point (500, 400)

```

## How Mouse Events Are Dispatched

All mouse actions in [`src/driver/pointer.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/driver/pointer.ts) ultimately route through CDP (`Input.dispatchMouseEvent`) for maximum fidelity. When the target page is backgrounded or CDP delivery fails, the system falls back to synthetic DOM events, ensuring that custom scrollers, map widgets, and drawing surfaces receive events indistinguishable from genuine user input.

The `wheel` helper specifically respects element visibility and focus states. If CDP injection is unavailable, it synthesizes a `WheelEvent` on the element under the specified point, maintaining compatibility with canvas-based zoom and pan controls.

## Summary

- **Canvas applications** lack addressable DOM nodes for internal elements, requiring coordinate-based interaction.
- **Three targeting modes** exist: pure selectors, absolute coordinates, and selector-plus-offset (recommended for canvases).
- **Source files** powering this workflow include [`src/driver/pointer.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/driver/pointer.ts) for the API, [`src/element-resolver.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/element-resolver.ts) for coordinate calculation, and [`src/format.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/format.ts) (line 477) for documentation.
- **Event delivery** uses CDP by default with synthetic event fallbacks for backgrounded tabs.
- **Practical applications** include clicking map tiles, drawing shapes via drag sequences, triggering canvas tooltips, and scrolling virtualized views.

## Frequently Asked Questions

### How do I click on a specific pixel inside a canvas element?

Pass an object containing both the `selector` and relative `x`, `y` coordinates to any mouse helper. For example, `{ selector: "canvas#game", x: 100, y: 50 }` clicks 100 pixels from the left edge and 50 pixels from the top of the canvas. The driver resolves the selector to get the canvas position, then adds your offset to calculate the absolute viewport coordinates.

### Can I drag across multiple points on a canvas using ego-browser?

Yes. The `drag` helper in [`src/driver/pointer.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/driver/pointer.ts) accepts an array of `MouseTarget` objects, allowing you to define complex paths. Each point in the array can use different targeting modes, though typically you’ll use the same canvas selector with varying `{x, y}` offsets to draw shapes or pan across maps.

### What happens if the page is backgrounded during canvas interaction?

When the browser page is not focused, CDP (`Input.dispatchMouseEvent`) may not deliver events reliably. In this case, ego-browser falls back to synthetic DOM events created via JavaScript injection. The `wheel` helper specifically checks visibility and synthesizes `WheelEvent` instances on elements under the target coordinates, ensuring canvas scrollers and custom widgets still receive input.

### Where are the mouse helper APIs defined in the ego-browser source code?

The primary mouse interaction APIs—including `click`, `drag`, `hover`, and `wheel`—are defined in [`src/driver/pointer.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/driver/pointer.ts). The `MouseTarget` type and resolution logic reside there as well, while [`src/element-resolver.ts`](https://github.com/citrolabs/ego-lite/blob/main/src/element-resolver.ts) handles the translation of selectors and offsets into concrete DOM rectangles and viewport coordinates.