# Action Retry Logic When Foreground Fallback Is Enabled in pi-computer-use

> Learn how pi-computer-use handles action retry logic with foreground fallback. Discover how background actions automatically retry in the foreground when needed for seamless operation.

- Repository: [injaneity/pi-computer-use](https://github.com/injaneity/pi-computer-use)
- Tags: deep-dive
- Published: 2026-07-16

---

**When background UI actions fail to produce visible changes in pi-computer-use, the library automatically retries eligible actions in the foreground using the `canRetryInForeground` predicate, while foreground-only actions bypass the background attempt entirely.**

The `pi-computer-use` library prioritizes non-intrusive automation by attempting UI interactions in the background using platform accessibility APIs. However, when an action does not produce the expected observable change, the action retry logic when foreground fallback is enabled ensures reliability by escalating to foreground delivery only when safe and necessary.

## How the Retry Mechanism Works in [`src/bridge.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/bridge.ts)

The core orchestration happens in `helperAct` within **src/bridge.ts**. This function implements a three-tier delivery strategy that balances stealth with reliability.

### Foreground-Only Action Bypass

Certain actions require immediate focus to function correctly. If the action has `action.needsForeground` or `action.usesCurrentFocus` set to true (such as `click` or `press` operations), `helperAct` immediately calls the platform backend with the `"foreground"` policy and returns the trace without attempting a background delivery.

```ts
// Example: Clicking bypasses background entirely
await pi.act({ action: "click", target: "@r2", params: { button: "left" } });

```

### Background Attempt and Initial Policy

For standard actions, `helperAct` first attempts delivery with the initial policy. In normal mode, this is `"background"`; in headless configurations, it uses `"ax_only"`. The code sends the request to the platform backend and awaits the outcome.

## The `canRetryInForeground` Decision Logic in [`src/actions.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/actions.ts)

After the background attempt completes, the library evaluates whether escalation is appropriate using the `canRetryInForeground` function defined in **src/actions.ts**:

```ts
export function canRetryInForeground(
    action: PreparedAction,
    outcome: "worked" | "didnt" | "unknown",
    headless: boolean,
): boolean {
    return !headless && outcome === "didnt"
        && (action.action === "typeText" || action.action === "keypress");
}

```

**Only `typeText` and `keypress` actions qualify for foreground retry**, and only when three conditions are met:
- The session is **not** running in headless mode (`!headless`)
- The background attempt returned `"didnt"` (indicating no observable UI change)
- The action type is explicitly whitelisted

This conservative approach ensures that only side-effect-free input actions get retried, preventing potentially disruptive operations from automatically stealing focus.

## Automatic Foreground Escalation and Trace Metadata

When `canRetryInForeground` returns true, `helperAct` re-executes the same request with the `"foreground"` policy and enriches the `ExecutionTrace` with detailed metadata:

- **`backgroundFirst: true`** – Indicates the background attempt occurred first
- **`escalatedToForeground: true`** – Confirms the fallback was triggered
- **`escalationReason: "side_effect_free_didnt"`** – Documents why escalation occurred
- **`backgroundAttempt`** – Contains the failed outcome details including the specific reason

### Error-Driven Fallback

If the platform backend throws a `"foreground_required"` error during the initial attempt (common when the Accessibility API detects a privileged operation), the `catch` block in `helperAct` performs an immediate foreground retry. This stores `outcome: "foreground_required"` as the escalation reason, ensuring the action succeeds even when the backend cannot determine foreground requirements ahead of time.

## Practical Code Examples

The following demonstrations show how the retry logic operates in practice:

```ts
// Background-first typing with potential foreground fallback
await pi.act({ action: "typeText", target: "@r1", params: { text: "hello" } });

```

In this scenario:
- If the background Accessibility API successfully injects the text, the trace reports `deliveryPolicy: "background"`
- If the target window shows no text change (outcome `"didnt"`), the library automatically retries in foreground mode
- The final trace includes `backgroundFirst: true` and `escalatedToForeground: true`

```ts
// Foreground-only execution (no retry logic)
await pi.act({ action: "click", target: "@button1", params: { button: "left" } });

```

Because click operations require current focus, `helperAct` immediately uses the `"foreground"` policy without attempting background delivery, ensuring the target element receives focus before the click occurs.

## Summary

- **Background preference**: `pi-computer-use` defaults to `"background"` delivery for minimal user disruption, using `"ax_only"` in headless mode
- **Selective retry**: Only `typeText` and `keypress` actions with a `"didnt"` outcome trigger foreground fallback via `canRetryInForeground`
- **Immediate foreground**: Actions with `needsForeground` bypass background attempts entirely
- **Rich telemetry**: Execution traces capture `escalatedToForeground`, `escalationReason`, and `backgroundAttempt` details for debugging
- **Error resilience**: `"foreground_required"` exceptions trigger automatic retry without requiring manual intervention

## Frequently Asked Questions

### What actions are eligible for foreground retry in pi-computer-use?

Only `typeText` and `keypress` actions are eligible for automatic foreground retry according to the `canRetryInForeground` implementation in [`src/actions.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/actions.ts). Actions like `click`, `press`, or `scroll` that set `needsForeground` to true bypass the background attempt entirely, while other actions that fail with a `"didnt"` outcome do not qualify for retry to prevent unwanted focus stealing.

### How does the library detect when a background action failed?

The platform backend returns an outcome status of `"worked"`, `"didnt"`, or `"unknown"` after each attempt. The retry logic specifically checks for `"didnt"` (indicating no observable UI change) combined with the action type whitelist. This precise detection ensures that `pi-computer-use` only escalates to foreground when the background approach definitively failed to produce effects.

### What metadata is added to the execution trace when fallback occurs?

When foreground fallback triggers, the `ExecutionTrace` object receives four key fields: `backgroundFirst: true` confirming the initial attempt, `escalatedToForeground: true` marking the escalation, `escalationReason: "side_effect_free_didnt"` (or `"foreground_required"` for error-driven cases), and a `backgroundAttempt` object containing the original failure details. These fields enable developers to audit when and why automation required foreground privileges.

### Does the retry logic work in headless mode?

No. The `canRetryInForeground` function explicitly checks `!headless` and returns false when running in headless mode. In headless configurations, `pi-computer-use` uses the `"ax_only"` delivery policy and does not attempt foreground escalation, as there is no interactive user session to disrupt or focus to steal.