# OCR Integration Through read_text for Non-Accessible Text in pi-computer-use

> Learn OCR integration with read_text in pi-computer-use. Extract text from images and UI elements using screenshots and external OCR for seamless data processing.

- Repository: [injaneity/pi-computer-use](https://github.com/injaneity/pi-computer-use)
- Tags: how-to-guide
- Published: 2026-07-16

---

**You can extract text from image-based or custom-drawn UI elements by capturing screenshots, applying external OCR, and feeding the results back into the `read_text` pipeline via synthetic output references.**

The `pi-computer-use` repository provides accessibility-driven automation tools that rely on native OS APIs to read text from UI elements. Because these APIs cannot expose custom-drawn canvas text or image-based dialogs, you must implement **OCR integration through `read_text` for non-accessible text** to capture content from inaccessible regions. This approach leverages the existing pagination system in [`src/output.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/output.ts) and the executor registration in [`src/bridge.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/bridge.ts) without requiring modifications to the core library.

## How read_text Handles Accessible vs. Non-Accessible Text

### Native Accessibility Limitations

The `read_text` tool operates on two reference types: **desktop (`@e`) refs** created by `inspect_ui` or `observe_ui`, and **browser (`@o`) refs** representing immutable session output. While `@e` refs expose the UI accessibility tree, they cannot surface content from non-reporting elements like canvas-rendered text or image-based CAPTCHAs. These limitations are inherent to the operating system's accessibility APIs, which only report explicitly tagged UI elements.

### The read_text Implementation

In [`src/bridge.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/bridge.ts), the executor registers the `read_text` handler that processes UTF-8 byte slices from captured outlines. The [`src/output.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/output.ts) module manages truncation for large outputs, adding a `continue` hint when pagination is required. This architecture is defined in [`src/contract.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/contract.ts) and documented in [`docs/usage.md`](https://github.com/injaneity/pi-computer-use/blob/main/docs/usage.md) and [`docs/architecture.md`](https://github.com/injaneity/pi-computer-use/blob/main/docs/architecture.md).

## OCR Integration Workflow for Non-Accessible Elements

When accessibility APIs fail to expose text, follow this three-step pattern:

1. **Capture a screenshot** of the target region using `observe_ui` with `screenshot:true` or the low-level `capture` API.
2. **Execute OCR** using an external library like Tesseract WASM or Google Vision.
3. **Store the result** as a synthetic `@o` ref using a `store_output` helper, enabling `read_text` to consume it with standard pagination.

The following example demonstrates basic `read_text` usage on an accessible element:

```typescript
// Obtain a reference to a text-bearing UI element
const { outline } = await pi.runTool("inspect_ui", { selector: "#article" });
const textRef = outline.refs[0];      // e.g. "@e3"

// Read the first 1 KB of text (UTF-8 bytes)
const { text, continue: next } = await pi.runTool("read_text", {
  ref: textRef,
  offset: 0,
  limit: 1024,
});
console.log(text);

// If output was truncated, request the next page
if (next) {
  const more = await pi.runTool("read_text", { ref: textRef, offset: 1024 });
  console.log(more.text);
}

```

For non-accessible regions, combine screenshot capture with OCR processing:

```typescript
// 1️⃣ Capture a screenshot of the target area
const { screenshot } = await pi.runTool("observe_ui", {
  selector: "#captcha",      // element that only shows an image
  screenshot: true,
});

// 2️⃣ Send PNG bytes to an OCR service (Tesseract WASM example)
import { recognize } from "tesseract-wasm";
const ocrResult = await recognize(screenshot.base64);

// 3️⃣ Store OCR text as synthetic output ref
const ocrRef = await pi.runTool("store_output", {
  content: ocrResult.text,
});

// 4️⃣ Read via read_text like any other ref
const { text } = await pi.runTool("read_text", { ref: ocrRef, offset: 0 });
console.log("OCR output:", text);

```

## Handling Large OCR Outputs with Pagination

When processing large images or extensive OCR results, implement chunking to respect memory constraints and API limits:

```typescript
const pageSize = 5000;
let offset = 0;
let fullText = "";

while (offset < screenshot.base64.length) {
  const chunk = screenshot.base64.slice(offset, offset + pageSize);
  const { text: part } = await recognize(chunk);
  fullText += part;
  offset += pageSize;
}

const ocrRef = await pi.runTool("store_output", { content: fullText });
const { text } = await pi.runTool("read_text", { ref: ocrRef, offset: 0 });
console.log(text);

```

## Key Source Files and Implementation Details

Understanding the core architecture helps you extend the library correctly:

- **[`src/bridge.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/bridge.ts)** – Registers the `read_text` executor and defines the tool payload structure.
- **[`src/output.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/output.ts)** – Handles truncation of large textual outputs and provides the `continue` hint for pagination.
- **[`src/contract.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/contract.ts)** – Declares the public `read_text` interface and parameter types.
- **[`docs/usage.md`](https://github.com/injaneity/pi-computer-use/blob/main/docs/usage.md)** – Documents pagination semantics and reference handling.
- **[`docs/architecture.md`](https://github.com/injaneity/pi-computer-use/blob/main/docs/architecture.md)** – Explains the immutable state model for UI roots and outlines.
- **[`extensions/computer-use.ts`](https://github.com/injaneity/pi-computer-use/blob/main/extensions/computer-use.ts)** – Declares tools for the Pi extension side; useful when adding custom helpers like `store_output`.

## Summary

- **OCR integration through `read_text`** bridges the gap when native accessibility APIs cannot expose visual content.
- Use `observe_ui` with `screenshot:true` to capture non-accessible regions as base64 PNGs.
- Process images with external OCR libraries, then store results via synthetic `@o` references.
- The existing pagination system in [`src/output.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/output.ts) handles large OCR outputs without core library modifications.
- All integration happens through the public tool contract in [`src/contract.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/contract.ts), requiring no changes to `pi-computer-use` internals.

## Frequently Asked Questions

### Can read_text directly perform OCR on screenshots?

No. The `read_text` tool only processes UTF-8 text from accessibility trees or stored output references. You must run OCR externally using libraries like Tesseract or Google Vision, then feed the resulting string back into the system via `store_output` to create a readable `@o` reference.

### What is the difference between @e and @o references?

`@e` (desktop) references point to live UI elements captured via `inspect_ui` or `observe_ui`, reading from the OS accessibility tree. `@o` (output) references are immutable session objects containing arbitrary text data, including synthetic OCR results you create with `store_output`. Both support the same `read_text` pagination semantics defined in [`src/contract.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/contract.ts).

### How does pagination work for large OCR results?

The `read_text` tool accepts `offset` and `limit` parameters to read specific byte ranges. When output exceeds the limit, [`src/output.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/output.ts) returns a `continue` flag. You increment the offset and call `read_text` again until the flag is false, as implemented in the executor within [`src/bridge.ts`](https://github.com/injaneity/pi-computer-use/blob/main/src/bridge.ts).

### Do I need to modify the core pi-computer-use library to add OCR support?

No. The architecture supports external integration through the public tool contract. You only need to implement a `store_output` helper in your extension (see [`extensions/computer-use.ts`](https://github.com/injaneity/pi-computer-use/blob/main/extensions/computer-use.ts) for patterns) to create synthetic references, then process screenshots with your chosen OCR library.