How to Use `search_ui` and `expand_ui` to Query Cached Immutable State in pi-computer-use

search_ui and expand_ui are read-only tools that operate on immutable, state-scoped UI outlines cached by stateId, allowing deterministic element lookup and selective expansion of truncated nodes without triggering live screenshots.

The pi-computer-use extension implements a state-scoped architecture where UI observation results are stored as immutable outlines. By leveraging these cached states, agents can query element hierarchies and reveal truncated subtrees using stable references (@eN) that remain consistent throughout the automation workflow.

Understanding the Immutable State Architecture

The extension manages UI interaction through request-local state objects defined in src/state.ts. When a tool call receives a stateId, it restores a saved outline rather than capturing a new screenshot. As documented in the architecture flow:

"find roots → observe one root → search/expand/inspect its state → act from that state"【https://github.com/injaneity/pi-computer-use/blob/main/docs/architecture.md#L3-L7】

Because the outline is immutable, element references (@eN) are guaranteed to point to the exact same UI elements for the lifetime of that state. This prevents race conditions and ensures that concurrent operations cannot interfere with each other. The cached outlines are also bounded, meaning large trees may have truncated nodes that require explicit expansion.

Querying Cached Outlines with search_ui

The search_ui tool performs pure cached-outline queries against the stored UI tree. Unless OCR escalation is triggered, it never interacts with the live interface.

In src/bridge.ts, the implementation follows this sequence:

1. Filter validation: The tool validates that at least one filter (text, role, or capability) is provided【https://github.com/injaneity/pi-computer-use/blob/main/src/bridge.ts#L17-L13】
2. State retrieval: Retrieves the current outline via currentOutlineOrThrow for the supplied stateId【https://github.com/injaneity/pi-computer-use/blob/main/src/bridge.ts#L8-L9】
3. Ranked matching: Executes searchOutlineRanked and collects up to 12 matches【https://github.com/injaneity/pi-computer-use/blob/main/src/bridge.ts#L14-L15】
4. Result formatting: Returns matched refs, roles, labels, and human-readable descriptions【https://github.com/injaneity/pi-computer-use/blob/main/src/bridge.ts#L33-L39】

OCR Escalation Behavior

If no matches are found and OCR is enabled, search_ui escalates to a live capture, merges the new outline into the cached state, and re-ranks the results【https://github.com/injaneity/pi-computer-use/blob/main/src/bridge.ts#L17-L30】. This ensures that text invisible to the initial accessibility tree can still be discovered without abandoning the cached state model.

// Search for a button in the cached state
const search = await pi.runTool("search_ui", {
  stateId: ui.stateId,
  text: "Submit",
  role: "button"
});
// Returns: { matches: [{ ref: "@e12", role: "button", label: "Submit" }] }

Expanding Truncated Nodes with expand_ui

When a node in the cached outline is truncated (platform omitted children to respect size limits) or its note region has changed, expand_ui retrieves a deeper view via a scoped live look.

The implementation in src/bridge.ts handles this workflow:

1. Ref validation: Verifies the ref argument is present【https://github.com/injaneity/pi-computer-use/blob/main/src/bridge.ts#L46-L48】
2. Node lookup: Locates the node using nodeByRef; throws an error if the ref is missing【https://github.com/injaneity/pi-computer-use/blob/main/src/bridge.ts#L48-L50】
3. Scoped observation: When the node is truncated, calls performLook to capture a live view of just that subtree, then merges it via graftScopedOutline【https://github.com/injaneity/pi-computer-use/blob/main/src/bridge.ts#L54-L66】
4. Budget folding: Returns the final view using foldToBudget with an optional depth limit【https://github.com/injaneity/pi-computer-use/blob/main/src/bridge.ts#L71-L73】

This approach keeps the operation lightweight by avoiding full-window re-observation while updating only the necessary portion of the immutable cache.

// Expand a truncated node to reveal its children
const expand = await pi.runTool("expand_ui", {
  stateId: ui.stateId,
  ref: "@e7",
  depth: 4  // Optional depth budget
});
// Returns the unfolded subtree under @e7

Complete Workflow Example

The canonical interaction pattern demonstrates how these tools maintain consistency across the automation lifecycle:

async function interactWithCachedState() {
  // 1. Establish the state boundary
  const roots = await pi.runTool("find_roots");
  const observed = await pi.runTool("observe_ui", { 
    ref: roots[0] 
  });  // Returns { stateId: "...", outline: ... }
  
  // 2. Query the immutable cache
  const search = await pi.runTool("search_ui", {
    stateId: observed.stateId,
    text: "Settings"
  });
  
  const targetRef = search.details?.matches?.[0]?.ref;
  
  // 3. Expand if truncated before acting
  const expanded = await pi.runTool("expand_ui", {
    stateId: observed.stateId,
    ref: targetRef,
    depth: 3
  });
  
  // 4. Act using the stable reference
  await pi.runTool("act_ui", {
    stateId: observed.stateId,
    actions: [{ action: "click", ref: targetRef }]
  });
}

All operations reference the same stateId, ensuring that @e12 in the search results refers to the identical element when passed to act_ui, even if the live UI has changed visually.

Summary

• search_ui performs read-only queries against cached UI outlines in src/state.ts, returning up to 12 ranked matches without live screenshots unless OCR escalation is required.
• expand_ui performs scoped live looks on truncated nodes via performLook and graftScopedOutline, revealing hidden children while preserving the original stateId.
• Both tools operate on immutable state where element references (@eN) remain stable for the duration of the session, as implemented in src/bridge.ts.
• The architecture follows a "cached-state-first" pattern: observe once, query repeatedly, expand selectively, and act deterministically.

Frequently Asked Questions

What is the difference between search_ui and observe_ui?

observe_ui captures a fresh accessibility tree from a live window and creates a new cached state, while search_ui only reads from an existing cached outline identified by stateId without interacting with the live UI. According to the source code in src/bridge.ts, search_ui uses currentOutlineOrThrow to retrieve stored state rather than initiating new platform observations.

When does expand_ui require a live screenshot?

expand_ui triggers a scoped live look only when the target node is marked as truncated or its note region has changed【https://github.com/injaneity/pi-computer-use/blob/main/src/bridge.ts#L54-L66】. If the node is already fully expanded in the cached outline, the tool returns the existing subtree immediately without capturing new screenshots.

How many elements can search_ui return?

The tool collects up to 12 matches from the ranked search results【https://github.com/injaneity/pi-computer-use/blob/main/src/bridge.ts#L14-L15】. If more elements match the criteria, only the top 12 by relevance score are returned, with an optional OCR escalation if zero matches are found and OCR mode is enabled.

Can I use search_ui and expand_ui with the same stateId concurrently?

Yes. Because the underlying outlines are immutable, multiple tool calls can safely read from and expand the same stateId without race conditions. The state management in src/state.ts ensures that each request-local state remains isolated, and graft operations in expand_ui create new immutable versions rather than mutating the existing cache.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}