# How to Use `search_ui` and `expand_ui` to Query Cached Immutable State in pi-computer-use

> Unlock efficient pi-computer-use querying with search_ui and expand_ui. Learn to access cached immutable state deterministically and expand UI nodes without live screenshots.

- Repository: [injaneity/pi-computer-use](https://github.com/injaneity/pi-computer-use)
- Tags: how-to-guide
- Published: 2026-07-16

---

**`search_ui` and `expand_ui` are read-only tools that operate on immutable, state-scoped UI outlines cached by `stateId`, allowing deterministic element lookup and selective expansion of truncated nodes without triggering live screenshots.**

The `pi-computer-use` extension implements a **state-scoped architecture** where UI observation results are stored as immutable outlines. By leveraging these cached states, agents can query element hierarchies and reveal truncated subtrees using stable references (`@eN`) that remain consistent throughout the automation workflow.

## Understanding the Immutable State Architecture

The extension manages UI interaction through **request-local state objects** defined in [`src/state.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/state.ts). When a tool call receives a `stateId`, it restores a saved outline rather than capturing a new screenshot. As documented in the architecture flow:

> "find roots → observe one root → **search/expand/inspect its state** → act from that state"【https://github.com/injaneity/pi-computer-use/blob/main/docs/architecture.md#L3-L7】

Because the outline is **immutable**, element references (`@eN`) are guaranteed to point to the exact same UI elements for the lifetime of that state. This prevents race conditions and ensures that concurrent operations cannot interfere with each other. The cached outlines are also **bounded**, meaning large trees may have truncated nodes that require explicit expansion.

## Querying Cached Outlines with `search_ui`

The `search_ui` tool performs **pure cached-outline queries** against the stored UI tree. Unless OCR escalation is triggered, it never interacts with the live interface.

In [`src/bridge.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/bridge.ts), the implementation follows this sequence:

1. **Filter validation**: The tool validates that at least one filter (`text`, `role`, or `capability`) is provided【https://github.com/injaneity/pi-computer-use/blob/main/src/bridge.ts#L17-L13】
2. **State retrieval**: Retrieves the current outline via `currentOutlineOrThrow` for the supplied `stateId`【https://github.com/injaneity/pi-computer-use/blob/main/src/bridge.ts#L8-L9】
3. **Ranked matching**: Executes `searchOutlineRanked` and collects up to **12** matches【https://github.com/injaneity/pi-computer-use/blob/main/src/bridge.ts#L14-L15】
4. **Result formatting**: Returns matched refs, roles, labels, and human-readable descriptions【https://github.com/injaneity/pi-computer-use/blob/main/src/bridge.ts#L33-L39】

### OCR Escalation Behavior

If no matches are found and OCR is enabled, `search_ui` escalates to a **live capture**, merges the new outline into the cached state, and re-ranks the results【https://github.com/injaneity/pi-computer-use/blob/main/src/bridge.ts#L17-L30】. This ensures that text invisible to the initial accessibility tree can still be discovered without abandoning the cached state model.

```typescript
// Search for a button in the cached state
const search = await pi.runTool("search_ui", {
  stateId: ui.stateId,
  text: "Submit",
  role: "button"
});
// Returns: { matches: [{ ref: "@e12", role: "button", label: "Submit" }] }

```

## Expanding Truncated Nodes with `expand_ui`

When a node in the cached outline is **truncated** (platform omitted children to respect size limits) or its note region has changed, `expand_ui` retrieves a deeper view via a scoped live look.

The implementation in [`src/bridge.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/bridge.ts) handles this workflow:

1. **Ref validation**: Verifies the `ref` argument is present【https://github.com/injaneity/pi-computer-use/blob/main/src/bridge.ts#L46-L48】
2. **Node lookup**: Locates the node using `nodeByRef`; throws an error if the ref is missing【https://github.com/injaneity/pi-computer-use/blob/main/src/bridge.ts#L48-L50】
3. **Scoped observation**: When the node is truncated, calls `performLook` to capture a live view of just that subtree, then merges it via `graftScopedOutline`【https://github.com/injaneity/pi-computer-use/blob/main/src/bridge.ts#L54-L66】
4. **Budget folding**: Returns the final view using `foldToBudget` with an optional depth limit【https://github.com/injaneity/pi-computer-use/blob/main/src/bridge.ts#L71-L73】

This approach keeps the operation lightweight by avoiding full-window re-observation while updating only the necessary portion of the immutable cache.

```typescript
// Expand a truncated node to reveal its children
const expand = await pi.runTool("expand_ui", {
  stateId: ui.stateId,
  ref: "@e7",
  depth: 4  // Optional depth budget
});
// Returns the unfolded subtree under @e7

```

## Complete Workflow Example

The canonical interaction pattern demonstrates how these tools maintain consistency across the automation lifecycle:

```typescript
async function interactWithCachedState() {
  // 1. Establish the state boundary
  const roots = await pi.runTool("find_roots");
  const observed = await pi.runTool("observe_ui", { 
    ref: roots[0] 
  });  // Returns { stateId: "...", outline: ... }
  
  // 2. Query the immutable cache
  const search = await pi.runTool("search_ui", {
    stateId: observed.stateId,
    text: "Settings"
  });
  
  const targetRef = search.details?.matches?.[0]?.ref;
  
  // 3. Expand if truncated before acting
  const expanded = await pi.runTool("expand_ui", {
    stateId: observed.stateId,
    ref: targetRef,
    depth: 3
  });
  
  // 4. Act using the stable reference
  await pi.runTool("act_ui", {
    stateId: observed.stateId,
    actions: [{ action: "click", ref: targetRef }]
  });
}

```

All operations reference the same `stateId`, ensuring that `@e12` in the search results refers to the identical element when passed to `act_ui`, even if the live UI has changed visually.

## Summary

- **`search_ui`** performs read-only queries against cached UI outlines in [`src/state.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/state.ts), returning up to 12 ranked matches without live screenshots unless OCR escalation is required.
- **`expand_ui`** performs **scoped live looks** on truncated nodes via `performLook` and `graftScopedOutline`, revealing hidden children while preserving the original `stateId`.
- Both tools operate on **immutable state** where element references (`@eN`) remain stable for the duration of the session, as implemented in [`src/bridge.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/bridge.ts).
- The architecture follows a "cached-state-first" pattern: observe once, query repeatedly, expand selectively, and act deterministically.

## Frequently Asked Questions

### What is the difference between `search_ui` and `observe_ui`?

`observe_ui` captures a fresh accessibility tree from a live window and creates a new cached state, while `search_ui` only reads from an existing cached outline identified by `stateId` without interacting with the live UI. According to the source code in [`src/bridge.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/bridge.ts), `search_ui` uses `currentOutlineOrThrow` to retrieve stored state rather than initiating new platform observations.

### When does `expand_ui` require a live screenshot?

`expand_ui` triggers a **scoped live look** only when the target node is marked as truncated or its note region has changed【https://github.com/injaneity/pi-computer-use/blob/main/src/bridge.ts#L54-L66】. If the node is already fully expanded in the cached outline, the tool returns the existing subtree immediately without capturing new screenshots.

### How many elements can `search_ui` return?

The tool collects up to **12 matches** from the ranked search results【https://github.com/injaneity/pi-computer-use/blob/main/src/bridge.ts#L14-L15】. If more elements match the criteria, only the top 12 by relevance score are returned, with an optional OCR escalation if zero matches are found and OCR mode is enabled.

### Can I use `search_ui` and `expand_ui` with the same `stateId` concurrently?

Yes. Because the underlying outlines are **immutable**, multiple tool calls can safely read from and expand the same `stateId` without race conditions. The state management in [`src/state.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/state.ts) ensures that each request-local state remains isolated, and graft operations in `expand_ui` create new immutable versions rather than mutating the existing cache.