Understanding worked/didnt/unknown Action Outcomes and Verification in pi-computer-use

The pi-computer-use framework categorizes every UI interaction into one of three outcomes—worked, didnt, or unknown—based on platform feedback, post-condition verification, and observed value checks defined in src/actions.ts.

The pi-computer-use library provides a deterministic way to automate desktop interactions by explicitly modeling action success states. When an agent performs a click, text entry, or keypress, the system doesn't just execute the command—it verifies whether the expected UI change actually occurred. This article explains how the three outcome states are defined in the source code and how the verification pipeline determines which state applies.

The Three Action Outcome States

Every action dispatched through the framework resolves to one of three mutually exclusive outcomes:

• worked — The agent performed the action and the expected UI change was observed. This occurs when the platform reports successful delivery or when post-condition verification confirms the change.
• didnt — The agent delivered the input but the UI did not change as expected. This happens when the platform returns a failure, a timeout occurs, or verification reports that the condition was not satisfied.
• unknown — The agent cannot determine whether the UI changed. This typically occurs in headless modes, when images are missing, or when the action's effect is not directly observable.

How Outcomes Are Determined in the Source Code

The outcome calculation logic lives primarily in src/actions.ts, where two core functions map raw platform results to the final three-state classification.

Mapping Verification Results with outcomeAfterCheck

The outcomeAfterCheck function (lines 16-20) maps verification step results onto the three outcomes. It accepts the current outcome and a check status, then returns the definitive result:

// src/actions.ts
export function outcomeAfterCheck(
    current: "worked" | "didnt" | "unknown",
    check: "verified" | "preexisting" | "failed"
): "worked" | "didnt" | "unknown" {
    if (check === "verified") return "worked";
    if (check === "failed") return "didnt";
    return current;
}

When the verification check returns "verified", the outcome becomes "worked". If it returns "failed", the outcome becomes "didnt". The "preexisting" status preserves the current outcome, allowing the system to maintain an "unknown" state when appropriate.

Validating Observed Values with outcomeAfterObservedValues

For setText actions specifically, the system performs an additional validation through outcomeAfterObservedValues (lines 22-30). This function checks whether the text values actually match the intended input:

// src/actions.ts
export function outcomeAfterObservedValues(
    current: "worked" | "didnt" | "unknown",
    actions: UiAction[],
    valueForRef: (ref: string) => string | undefined,
): "worked" | "didnt" | "unknown" {
    if (actions.length === 0 ||
        actions.some((a) => a.action !== "setText" || !a.ref)) return current;
    const matches = actions.every(
        (a) => valueForRef(a.ref!) === (a.text ?? "")
    );
    return matches ? "worked" : current;
}

This helper ensures that when you set text on multiple fields, the outcome only becomes "worked" if every field contains the exact value requested.

The Action Execution Pipeline

The complete lifecycle of an action outcome spans multiple stages across the codebase.

Step-by-Step Outcome Assembly

According to the implementation in performDesktopTransaction (lines 76-130 of src/actions.ts), the system assembles outcomes through this sequence:

1. Prepare the action via prepareUiAction → prepareAction
2. Dispatch through helperAct (foreground or background delivery)
3. Collect a trace using executionTraceFromAct and aggregateExecutions, containing the raw platform outcome (result.outcome)
4. Optional verification — if an expect clause exists, call the platform waitFor helper to produce verification.status (verified, preexisting, or failed)
5. Apply outcomeAfterCheck to combine the raw outcome with the verification result
6. Apply outcomeAfterObservedValues to incorporate value-based evidence from setText operations

Foreground Retry Escalation

Some actions can escalate from background (stealth) delivery to foreground when the initial outcome is "didnt". The canRetryInForeground function (lines 12-15) defines this logic:

// src/actions.ts
export function canRetryInForeground(
    action: PreparedAction,
    outcome: "worked" | "didnt" | "unknown",
    headless: boolean
): boolean {
    return !headless && outcome === "didnt" &&
           (action.action === "typeText" || action.action === "keypress");
}

Currently, only typeText and keypress actions support this escalation. The retry logic lives in helperAct within src/bridge.ts, where the system checks if (canRetryInForeground(...)) before attempting a foreground delivery.

Verification Flow for expect Conditions

When a transaction includes an expect block, performDesktopTransaction performs additional validation:

1. Call the platform waitFor API with the specified UI condition (element reference, text match, or disappearance)
2. Interpret the returned verification object:
   • found → condition appeared (or disappeared if gone is specified)
   • timedOut → condition never appeared
3. Map these states to verified, preexisting, or failed
4. Apply outcomeAfterCheck to determine the final outcome

This verification code appears in the conditional block starting at line 90 of src/actions.ts.

Practical Examples

Example 1: Click with Post-Condition Verification

import { act_ui } from "pi-computer-use";

await act_ui({
  stateId: "<previous-observe-id>",
  expect: {
    ref: "@r2",
    text: "Submit",
    timeoutMs: 3000
  },
  actions: [
    { action: "click", ref: "@r2" }
  ]
});

This example targets a button with reference @r2. If the click succeeds and the button text remains "Submit", the trace ends with outcome: "worked". If the button disappears or the text changes unexpectedly, the outcome becomes "didnt". If running in headless mode where the button text cannot be read, the outcome remains "unknown".

Example 2: Text Entry with Foreground Retry

await act_ui({
  stateId: "<observe-id>",
  actions: [
    {
      action: "typeText",
      ref: "@r5",
      text: "Hello world"
    }
  ]
});

Here, the action first attempts background delivery. If the platform returns outcome: "didnt" (for example, if the field never gained focus), canRetryInForeground triggers a foreground retry. The final outcome reflects the foreground attempt result.

Example 3: Batch Actions with Observed Value Checking

await act_ui({
  stateId: "<observe-id>",
  actions: [
    { action: "setText", ref: "@r7", text: "foo" },
    { action: "setText", ref: "@r8", text: "bar" }
  ]
});

After executing this batch, outcomeAfterObservedValues reads the current values of @r7 and @r8. If both match the requested text exactly, the overall outcome becomes "worked". If either value differs, the outcome remains whatever the raw platform reported (often "unknown").

Summary

• Three outcomes (worked, didnt, unknown) provide deterministic feedback on whether UI interactions achieved their intended effects.
• outcomeAfterCheck maps verification results to final states, converting "verified" to "worked" and "failed" to "didnt".
• outcomeAfterObservedValues specifically validates setText operations by comparing actual field contents against requested values.
• Foreground retry automatically escalates failed "didnt" outcomes for typeText and keypress actions when running in headed mode.
• The complete logic resides in src/actions.ts, with platform integration handled in src/bridge.ts and runtime state managed in src/runtime.ts.

Frequently Asked Questions

What is the difference between "didnt" and "unknown" outcomes?

The "didnt" outcome indicates that the system successfully delivered the input to the application, but the expected UI change did not occur—such as when a click is sent but a button doesn't trigger its action. The "unknown" outcome indicates that the system cannot determine whether the change occurred, typically due to environmental limitations like headless execution or missing screenshots where the UI state cannot be observed.

When does the system automatically retry actions in the foreground?

According to canRetryInForeground in src/actions.ts, the system only retries actions when the outcome is "didnt", the session is not headless, and the action type is either "typeText" or "keypress". This escalation attempts to resolve focus-related failures by bringing the target window to the foreground before retrying the input delivery.

How does the expect clause affect the final outcome determination?

When you include an expect block in your action call, performDesktopTransaction calls the platform's waitFor API to verify the post-condition. The verification result (verified, preexisting, or failed) is then passed to outcomeAfterCheck, which has the final authority to promote an outcome to "worked" (if verified) or demote it to "didnt" (if failed), overriding the raw platform result.

Why do setText actions receive special observed value checking?

setText actions modify application state by setting field values, which creates an observable property that can be verified independently of the platform's execution report. The outcomeAfterObservedValues function exploits this by reading the actual text content from the UI elements after the action completes, providing concrete evidence that the operation succeeded rather than relying solely on the platform's delivery confirmation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}