How the Cursor Overlay Animation Works Without Blocking Action Delivery in pi-computer-use

The cursor overlay animation operates as a purely visual effect that runs independently of the action-delivery pipeline, using a backend flag, native helper dispatch, and a non-modal SwiftUI window to ensure zero interference with high-throughput automation.

The pi-computer-use framework implements a ghost cursor visualization that helps developers debug automation sequences without degrading performance. Understanding how this cursor overlay animation works without blocking action delivery requires examining the three-layer architecture that keeps visual feedback strictly separate from input execution.

Architecture of the Non-Blocking Overlay

The system achieves zero blocking through three tightly-coupled components that communicate via metadata flags rather than synchronous waits.

Backend Flag System

In src/platform/macos/backend.ts (lines 15-22), the TypeScript backend adds a cursorOverlay Boolean to every act or actBatch request. This field merely indicates whether the native helper should trigger the visual effect after completing the accessibility action. Because the flag is a simple metadata field, the helper processes the request immediately without waiting for any animation to complete.

Native Helper Bridge

The native/macos/bridge.swift file (line 1888) receives the cursorOverlay flag and executes the real macOS Accessibility (AX) action on the main thread. Once the AX call returns, the helper checks the flag. If true, it forwards the target screen point to the cursor renderer via AgentCursor.shared.animate(to:cursorPoint). This handoff occurs after the accessibility action completes, ensuring the visual effect never delays the return of the HelperActResult.

SwiftUI Rendering Layer

The visual implementation resides in two Swift files that operate entirely outside the input pipeline:

• native/macos/agent_cursor.swift (lines 14-35, 51-62): Defines AgentCursorOverlayWindow with ignoresMouseEvents = true and canBecomeKey = false, creating a borderless, click-through window that cannot receive input.
• native/macos/agent_cursor_motion.swift (lines 5-30): Implements the AgentCursorRenderer using a Dubins-path motion model and a TimelineView refreshing at 120Hz.

The animation runs on the main actor via a background Task that updates the renderer’s tick method, while the TimelineView handles asynchronous repaints. This dual mechanism ensures the overlay updates smoothly without stalling the helper’s socket communication.

Why Action Delivery Never Blocks

Four specific design decisions ensure the cursor overlay animation remains strictly observational:

1. Synchronous action execution first: The native helper always performs the required UI interaction and returns its result before considering the overlay.
2. Fire-and-forget animation scheduling: The animate(to:) method schedules the visual transition and returns instantly, making the overlay a pure consumer of the action result.
3. Non-modal window properties: As configured in lines 51-62 of agent_cursor.swift, the overlay window cannot become key or receive mouse events, isolating it from the input pipeline.
4. Cancellable idle hide task: Lines 29-34 of agent_cursor.swift implement an idleHideTask that schedules overlay disappearance after 8 seconds, automatically cancelling and rescheduling on subsequent animations to prevent main thread stalls.

Configuration and Usage

Enable the overlay globally through environment variables or configuration files, then control it per-request as needed.

Enabling the Overlay Globally

Set the environment variable before launching your application:

PI_COMPUTER_USE_CURSOR_OVERLAY=true node myApp.js

Or check the configuration programmatically in src/config.ts:

import { getComputerUseConfig } from "./config.ts";

if (getComputerUseConfig().cursor_overlay) {
  console.log("Ghost cursor overlay is enabled");
}

Per-Request Control

The default configuration applies to all actions, but you can disable the overlay for specific calls by passing cursorOverlay: false:

import { macosBackend } from "./platform/macos/backend.ts";

// This action will show the overlay (if globally enabled)
await macosBackend.act({
  target: { focus: { pid: 1234 } },
  type: "click",
  params: { x: 400, y: 300 },
});

// This action explicitly disables the overlay
await macosBackend.act({
  target: { focus: { pid: 1234 } },
  type: "click",
  params: { x: 400, y: 300 },
  cursorOverlay: false,
});

Summary

• The cursor overlay animation in pi-computer-use operates through a flag-based system where src/platform/macos/backend.ts signals intent without blocking.
• Native execution in native/macos/bridge.swift completes accessibility actions before triggering visual effects, ensuring HelperActResult returns immediately.
• SwiftUI rendering uses a click-through window (ignoresMouseEvents = true) and asynchronous TimelineView updates to avoid input pipeline interference.
• The Dubins-path motion model and background Task scheduling allow 120Hz animation without stalling the main thread or socket communication.
• Developers can enable the overlay globally via PI_COMPUTER_USE_CURSOR_OVERLAY or disable it per-request using the cursorOverlay: false parameter.

Frequently Asked Questions

Does the overlay capture mouse events or interfere with system input?

No. The AgentCursorOverlayWindow is explicitly configured with ignoresMouseEvents = true and canBecomeKey = false in native/macos/agent_cursor.swift (lines 51-62). This creates a completely passive visualization layer that cannot receive clicks, keyboard input, or focus changes, ensuring your automation continues to interact with the actual system cursor.

Can I customize the animation speed or path algorithm?

The animation uses a Dubins-path motion model implemented in native/macos/agent_cursor_motion.swift (lines 5-30) with a TimelineView refreshing at 120Hz. While the repository provides smooth motion through these defaults, the motion model parameters and timing logic reside in the AgentCursorRenderer class, allowing modification of the path calculation and update intervals if you fork the native Swift code.

Does this work on non-macOS platforms?

The non-blocking cursor overlay architecture described here is specific to the macOS implementation found in src/platform/macos/backend.ts and the native/macos/ directory. Other platforms in the pi-computer-use repository may implement different visualization strategies or may not provide this specific ghost cursor feature, as it relies on macOS-specific NSWindow and SwiftUI capabilities.

How does the idle hide mechanism work without blocking subsequent actions?

The idleHideTask defined in native/macos/agent_cursor.swift (lines 29-34) uses Swift's structured concurrency to schedule a cancellation-enabled task that hides the overlay after 8 seconds of inactivity. Because each new animation call cancels the previous task and reschedules a new one, the mechanism never accumulates blocking operations or delays the processing of incoming act requests, maintaining the non-blocking guarantee even during rapid sequences of actions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}