How the Pointer and Keyboard Drivers in Ego-Browser Implement Click, Fill, and Press Interactions

Ego-Browser's driver layer translates high-level automation helpers into Chrome DevTools Protocol (CDP) commands, using PointerDriver for mouse actions and KeyboardDriver for key events to execute reliable clicks, text fills, and key presses.

The citrolabs/ego-lite repository provides a lightweight browser automation framework where user-facing helpers like click, fill, and press are backed by specialized driver classes. Understanding how these pointer and keyboard drivers in ego-browser convert declarative commands into precise CDP messaging reveals the architecture behind stable, fast browser automation.

Pointer Driver Architecture

The PointerDriver class in src/driver/pointer.ts handles all mouse-based interactions. It bridges high-level element targeting with low-level CDP input dispatching.

Element Location and Resolution

Before any interaction, the driver resolves locators (CSS selectors, ARIA roles, or XPath expressions) through the element-resolver module. This resolution yields two critical pieces of data: the backendNodeId for DOM identification and the element's bounding rectangle geometry. The driver calculates the center coordinates (x, y) from this bounding box to ensure interactions target the middle of the element, avoiding off-center misses.

Mouse Event Synthesis for Clicks

For a standard click operation, the driver synthesizes a complete mouse gesture using Input.dispatchMouseEvent. It sends paired events: first mousePressed, then mouseReleased, both targeting the center coordinates derived from the element's bounding box. The method signature includes precise parameters:

await dispatchMouseEvent('mousePressed', {
  x: centerX,
  y: centerY,
  button: 'left',
  clickCount: 1
})

The button: 'left' and clickCount: 1 parameters ensure the CDP command matches standard user behavior, while the exact x/y coordinates guarantee the click registers on the intended element even if the DOM has shifted since location.

Text Input Handling (Fill)

The fill operation combines pointer and keyboard techniques. First, the driver ensures focus by dispatching mousePressed and mouseReleased events at the input element's center. Then it attempts the fast path using Input.insertText, passing the complete string to the browser in a single command.

If insertText is unavailable (older CDP versions) or fails, the driver falls back to a per-character loop using Input.dispatchKeyEvent. This fallback handles modifier logic internally, automatically pressing Shift for capital letters and special characters to maintain correct casing.

Error Handling and Retries

When an element is not ready—perhaps still loading or detached from the DOM—the driver throws an ElementResolutionError marked as transient. This flag signals higher-level "wait-until-stable" loops to retry the operation, preventing flaky tests without requiring explicit sleep statements in user code.

Keyboard Driver Implementation

The KeyboardDriver in src/driver/keyboard.ts manages key events and text entry through the CDP Input domain.

Key Mapping and CDP Descriptors

The driver maintains an internal map from human-readable key names ("Enter", "ArrowUp", "Tab") to Chrome-specific CDP key descriptors. Each descriptor contains the code, key, windowsVirtualKeyCode, and nativeVirtualKeyCode required by the CDP specification. This mapping stays synchronized with the Chrome UI Events spec to ensure cross-platform consistency.

Press Implementation

A simple press operation (e.g., press('Enter')) generates two discrete CDP commands:

// Key down
await dispatchKeyEvent('keyDown', { code: 'Enter', key: 'Enter' })

// Key up
await dispatchKeyEvent('keyUp', { code: 'Enter', key: 'Enter' })

For modifier combinations (Shift+Enter, Ctrl+A), the driver accepts an options object and sets the modifiers bitmask on both events before dispatching.

Text Entry Fallbacks

When the higher-level type helper is invoked, the keyboard driver mirrors the pointer driver's logic: it prefers Input.insertText for speed, but falls back to individual dispatchKeyEvent calls when necessary. This dual-path approach balances performance (bulk text insertion) with compatibility (sites that listen for individual keystrokes).

Synchronization Strategies

After each key event, the driver optionally waits for a brief idle period, allowing the browser's event loop to process JavaScript handlers attached to the input. This prevents race conditions where subsequent commands execute before onChange or onKeyUp handlers complete, a common source of instability in fast automation scripts.

CDP Execution Layer

Both drivers rely on a shared CDP executor defined in src/cdp-eval.ts. This wrapper handles raw CDP socket communication, adds structured logging for debugging, and implements automatic retries on transient transport errors. It also respects the global timeout configuration, ensuring that stuck commands fail gracefully rather than hanging indefinitely.

Practical Usage Examples

These driver functions power the public helper API. Here is how high-level calls translate to driver actions:

// Click resolves the element, then dispatches mousePressed/mouseReleased
await click('loc=css:#submit')

// Fill focuses via click, then inserts text via insertText or key events
await fill('loc=css:#search', 'ego-browser')

// Press sends keyDown/keyUp pairs
await press('Enter')

For custom tooling that requires explicit control over the CDP layer:

import { resolveLocator } from './element-resolver.js'
import { dispatchMouseEvent, insertText, dispatchKeyEvent } from './cdp-eval.js'

const elem = await resolveLocator('loc=css:#name')
const { x, y } = elem.center

// Manual click sequence
await dispatchMouseEvent('mousePressed', { x, y, button: 'left' })
await dispatchMouseEvent('mouseReleased', { x, y, button: 'left' })

// Direct text insertion
await insertText('Alice')

// Manual Tab key press
await dispatchKeyEvent('keyDown', { code: 'Tab', key: 'Tab' })
await dispatchKeyEvent('keyUp', { code: 'Tab', key: 'Tab' })

Summary

• src/driver/pointer.ts implements PointerDriver, handling element resolution, synthesized mouse events via Input.dispatchMouseEvent, and transient error recovery for clicks and focus actions.
• src/driver/keyboard.ts implements KeyboardDriver, mapping human-readable keys to CDP descriptors and dispatching Input.dispatchKeyEvent pairs for presses, with modifier support and synchronization delays.
• Text input uses Input.insertText as a fast path, falling back to per-character dispatchKeyEvent loops for compatibility.
• Both drivers utilize src/cdp-eval.ts for robust CDP communication, logging, and retry logic.
• The architecture allows pure async helper functions that compose cleanly in automation scripts without exposing low-level protocol details.

Frequently Asked Questions

How does ego-browser calculate where to click on an element?

The pointer driver calls the element-resolver to obtain the target's bounding rectangle, then calculates the center coordinates (x, y). These exact coordinates are passed to Input.dispatchMouseEvent with button: 'left' and clickCount: 1, ensuring the click hits the center of the element regardless of viewport scrolling.

What happens if insertText fails when filling a form?

If Input.insertText is unavailable or rejected, the pointer driver automatically falls back to a series of Input.dispatchKeyEvent calls, sending one event per character. This fallback handles modifier keys internally (e.g., pressing Shift for uppercase letters) to ensure accurate text entry on sites that don't support the direct text insertion method.

Can I combine modifier keys like Ctrl or Shift with press commands?

Yes. The keyboard driver accepts a modifiers option that sets a bitmask for Shift, Ctrl, Alt, and Meta. When you pass these options to press(), the driver includes the modifier flags in both the keyDown and keyUp events dispatched via Input.dispatchKeyEvent, enabling combinations like Ctrl+A or Shift+Enter.

Why do the drivers throw ElementResolutionError instead of waiting automatically?

The drivers throw ElementResolutionError marked as transient to separate concerns: the driver layer focuses on immediate execution, while higher-level helper contexts implement "wait-until-stable" retry loops. This design allows the same driver code to work in both strict immediate-mode scripts and resilient polling-based wait strategies without coupling the timing logic to the CDP interaction layer.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}