How act_ui Transactions Manage Preconditions and Postconditions with expect in pi-computer-use

The act_ui tool uses the expect parameter to enforce preconditions before executing UI actions and verify postconditions afterward, ensuring deterministic automation through a bidirectional wait-for polling mechanism.

The act_ui transaction system in the pi-computer-use repository provides robust UI automation by validating application state at critical execution points. By leveraging the optional expect field in request payloads, agents can define precise UiCondition criteria that must be satisfied both before and after action sequences. This pattern eliminates race conditions in asynchronous desktop environments and provides clear failure semantics when UI states deviate from expectations.

Understanding the expect Parameter Structure

The expect property operates as a declarative condition descriptor defined in src/contract.ts as the UiCondition type (line 82). This structure supports element identification through text, role, and value attributes, alongside behavioral flags that control condition evaluation logic.

Core Condition Attributes

The UiCondition interface specifies several key fields that drive the validation engine:

• text, role, value: Attributes used to locate target UI elements within the accessibility tree
• gone: When set to true, inverts the condition to wait for element absence rather than presence
• scopeRef: References a previously captured element to constrain the search context
• scopeExact: Enforces strict matching within the scoped context
• timeoutMs: Overrides the default polling timeout for condition satisfaction

Precondition Validation Before Execution

Before any UI interactions occur, the bridge module validates and enforces preconditions through a dedicated polling mechanism defined in src/bridge.ts.

Condition Parsing with validateCondition

The validateCondition helper (line 1514 in src/bridge.ts) normalizes incoming expect parameters by mapping generic roles to platform-specific accessibility values and constructing canonical condition objects. This normalization ensures consistent behavior across different operating system APIs while validating that the condition structure conforms to the UiCondition contract.

The Wait-for Polling Loop

The bridge executes a wait-for loop that polls the UI state until the expect condition becomes satisfied or the timeout expires. When scopeRef is provided, the search context narrows to a previously captured element reference, enabling fine-grained synchronization for dependent multi-step workflows. If expect.gone is active, the loop continues until the specified element disappears from the accessibility tree.

Action Execution and Postcondition Verification

Once preconditions are satisfied, the system executes the transaction body and immediately validates the outcome.

Sequential Action Processing

The performAct function processes the actions array—containing UiAction objects such as clicks, keystrokes, and navigation commands—within a sandboxed runtime environment. If any action fails, the transaction aborts immediately without proceeding to postcondition checks, preventing cascading failures in uncertain UI states.

Postcondition Re-evaluation

After successful action completion, the same expect condition is re-evaluated to verify the UI reached the desired terminal state. This postcondition check compares the actual UI state against the expected criteria; mismatches generate detailed error messages containing the specific condition details that failed verification. This bidirectional validation ensures that transient states during action execution resolve into the expected final configuration.

Error Handling and State Propagation

All condition-related failures—including timeouts, mismatched roles, and missing elements—throw standard Error objects that bubble up to the agent's tool executor. The executeAct wrapper catches these exceptions and returns structured AgentToolResult objects that preserve the original expect description for debugging purposes. The StaleResourceStateError class in src/runtime.ts specifically handles cases where state preconditions fall out of sync during execution, providing granular error classification for recovery logic.

Practical Implementation Examples

The following TypeScript examples demonstrate practical usage patterns for the expect parameter in act_ui transactions:

// Click a button and wait until a status label shows "Done"
await executeAct({
  stateId: "mySession",
  expect: {               // pre-condition: button must be present
    text: "Start",
    role: "button"
  },
  actions: [
    { type: "click", selector: { text: "Start", role: "button" } },
    // post-condition will be re-checked automatically:
    // we expect the label "Done" to appear after the click
    { type: "wait", expect: { text: "Done", role: "label" } }
  ]
});

// Type into a field, then verify the modal disappears
await executeAct({
  stateId: "editSession",
  expect: {               // pre-condition: modal must be visible
    text: "Edit item",
    role: "dialog"
  },
  actions: [
    { type: "type", selector: { text: "Edit item", role: "dialog" }, text: "New name" },
    { type: "click", selector: { text: "Save", role: "button" } }
  ],
  // post-condition: the dialog should be gone
  expect: { gone: true, text: "Edit item", role: "dialog" }
});

Summary

• The expect parameter in act_ui transactions serves as both a precondition gate and postcondition validator, creating a reliable execution boundary
• src/contract.ts defines the UiCondition structure supporting text, role, value, and the gone flag for absence detection
• validateCondition in src/bridge.ts normalizes conditions and handles platform-specific role mapping before execution
• Precondition polling waits for element presence or absence before executing actions, with optional scoping via scopeRef
• Postcondition verification ensures the UI reaches the expected final state after action completion
• Errors propagate through AgentToolResult objects with preserved condition metadata, while StaleResourceStateError handles synchronization failures

Frequently Asked Questions

What happens if the expect precondition times out?

If the condition specified in expect fails to materialize within timeoutMs (or the default timeout), the bridge throws an Error that propagates to executeAct, returning a failed AgentToolResult without executing any UI actions. The error message includes the specific condition details that timed out, enabling precise debugging of synchronization failures.

Can expect be used to wait for elements to disappear?

Yes. Setting expect.gone to true inverts the condition logic, causing both precondition and postcondition checks to wait for the element's absence rather than presence. This pattern is essential for verifying modal dismissal, loading indicator removal, or transition completion in modern UI workflows.

How does scopeRef improve transaction reliability?

The scopeRef field constrains condition polling to a previously captured element reference rather than the entire document accessibility tree. This scoping reduces false positives from similar elements elsewhere in the UI and improves polling performance when monitoring specific regions, particularly in complex applications with dynamic content.

Where is the act_ui tool registered for agent use?

The tool schema and registration occur in extensions/computer-use.ts, which exposes the expect parameter to OpenAI function-calling agents and maps incoming requests to the bridge implementation in src/bridge.ts. This file defines the JSON schema that agents use to construct valid act_ui requests with properly formatted condition objects.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}