Action Retry Logic When Foreground Fallback Is Enabled in pi-computer-use

When background UI actions fail to produce visible changes in pi-computer-use, the library automatically retries eligible actions in the foreground using the canRetryInForeground predicate, while foreground-only actions bypass the background attempt entirely.

The pi-computer-use library prioritizes non-intrusive automation by attempting UI interactions in the background using platform accessibility APIs. However, when an action does not produce the expected observable change, the action retry logic when foreground fallback is enabled ensures reliability by escalating to foreground delivery only when safe and necessary.

How the Retry Mechanism Works in src/bridge.ts

The core orchestration happens in helperAct within src/bridge.ts. This function implements a three-tier delivery strategy that balances stealth with reliability.

Foreground-Only Action Bypass

Certain actions require immediate focus to function correctly. If the action has action.needsForeground or action.usesCurrentFocus set to true (such as click or press operations), helperAct immediately calls the platform backend with the "foreground" policy and returns the trace without attempting a background delivery.

// Example: Clicking bypasses background entirely
await pi.act({ action: "click", target: "@r2", params: { button: "left" } });

Background Attempt and Initial Policy

For standard actions, helperAct first attempts delivery with the initial policy. In normal mode, this is "background"; in headless configurations, it uses "ax_only". The code sends the request to the platform backend and awaits the outcome.

The canRetryInForeground Decision Logic in src/actions.ts

After the background attempt completes, the library evaluates whether escalation is appropriate using the canRetryInForeground function defined in src/actions.ts:

export function canRetryInForeground(
    action: PreparedAction,
    outcome: "worked" | "didnt" | "unknown",
    headless: boolean,
): boolean {
    return !headless && outcome === "didnt"
        && (action.action === "typeText" || action.action === "keypress");
}

Only typeText and keypress actions qualify for foreground retry, and only when three conditions are met:

• The session is not running in headless mode (!headless)
• The background attempt returned "didnt" (indicating no observable UI change)
• The action type is explicitly whitelisted

This conservative approach ensures that only side-effect-free input actions get retried, preventing potentially disruptive operations from automatically stealing focus.

Automatic Foreground Escalation and Trace Metadata

When canRetryInForeground returns true, helperAct re-executes the same request with the "foreground" policy and enriches the ExecutionTrace with detailed metadata:

• backgroundFirst: true – Indicates the background attempt occurred first
• escalatedToForeground: true – Confirms the fallback was triggered
• escalationReason: "side_effect_free_didnt" – Documents why escalation occurred
• backgroundAttempt – Contains the failed outcome details including the specific reason

Error-Driven Fallback

If the platform backend throws a "foreground_required" error during the initial attempt (common when the Accessibility API detects a privileged operation), the catch block in helperAct performs an immediate foreground retry. This stores outcome: "foreground_required" as the escalation reason, ensuring the action succeeds even when the backend cannot determine foreground requirements ahead of time.

Practical Code Examples

The following demonstrations show how the retry logic operates in practice:

// Background-first typing with potential foreground fallback
await pi.act({ action: "typeText", target: "@r1", params: { text: "hello" } });

In this scenario:

• If the background Accessibility API successfully injects the text, the trace reports deliveryPolicy: "background"
• If the target window shows no text change (outcome "didnt"), the library automatically retries in foreground mode
• The final trace includes backgroundFirst: true and escalatedToForeground: true

// Foreground-only execution (no retry logic)
await pi.act({ action: "click", target: "@button1", params: { button: "left" } });

Because click operations require current focus, helperAct immediately uses the "foreground" policy without attempting background delivery, ensuring the target element receives focus before the click occurs.

Summary

• Background preference: pi-computer-use defaults to "background" delivery for minimal user disruption, using "ax_only" in headless mode
• Selective retry: Only typeText and keypress actions with a "didnt" outcome trigger foreground fallback via canRetryInForeground
• Immediate foreground: Actions with needsForeground bypass background attempts entirely
• Rich telemetry: Execution traces capture escalatedToForeground, escalationReason, and backgroundAttempt details for debugging
• Error resilience: "foreground_required" exceptions trigger automatic retry without requiring manual intervention

Frequently Asked Questions

What actions are eligible for foreground retry in pi-computer-use?

Only typeText and keypress actions are eligible for automatic foreground retry according to the canRetryInForeground implementation in src/actions.ts. Actions like click, press, or scroll that set needsForeground to true bypass the background attempt entirely, while other actions that fail with a "didnt" outcome do not qualify for retry to prevent unwanted focus stealing.

How does the library detect when a background action failed?

The platform backend returns an outcome status of "worked", "didnt", or "unknown" after each attempt. The retry logic specifically checks for "didnt" (indicating no observable UI change) combined with the action type whitelist. This precise detection ensures that pi-computer-use only escalates to foreground when the background approach definitively failed to produce effects.

What metadata is added to the execution trace when fallback occurs?

When foreground fallback triggers, the ExecutionTrace object receives four key fields: backgroundFirst: true confirming the initial attempt, escalatedToForeground: true marking the escalation, escalationReason: "side_effect_free_didnt" (or "foreground_required" for error-driven cases), and a backgroundAttempt object containing the original failure details. These fields enable developers to audit when and why automation required foreground privileges.

Does the retry logic work in headless mode?

No. The canRetryInForeground function explicitly checks !headless and returns false when running in headless mode. In headless configurations, pi-computer-use uses the "ax_only" delivery policy and does not attempt foreground escalation, as there is no interactive user session to disrupt or focus to steal.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}