# How to Implement a Custom Operator Class for UI-TARS SDK

> Learn how to implement a custom Operator class for the UI-TARS SDK. Extend the abstract Operator class and implement screenshot and execute methods for custom UI automation.

- Repository: [Bytedance Inc./UI-TARS-desktop](https://github.com/bytedance/UI-TARS-desktop)
- Tags: how-to-guide
- Published: 2026-05-10

---

**To implement a custom Operator class for the UI-TARS SDK, extend the abstract `Operator` class and implement the `screenshot()` method to capture UI state and the `execute()` method to handle parsed model predictions.**

The UI-TARS SDK, part of the `bytedance/UI-TARS-desktop` repository, provides a flexible framework for building GUI automation agents. The `Operator` abstraction serves as the critical bridge between the AI model's predictions and your specific UI environment, requiring you to implement screenshot capture and action execution logic tailored to your application.

## Understanding the Operator Contract

The UI-TARS SDK defines the operator interface in [`packages/ui-tars/sdk/src/types.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/packages/ui-tars/sdk/src/types.ts) (lines 66-73). According to the source code, every operator must satisfy two fundamental requirements:

- **Screenshot capture**: The `screenshot()` method must return a `Promise<ScreenshotOutput>` containing a base64-encoded image string, width, height, and screen scale factor.
- **Action execution**: The `execute()` method receives an `ExecuteParams` object and returns a `Promise<ExecuteOutput>` with a status of either `StatusEnum.RUNNING` or `StatusEnum.END`.

The abstract class extends `BaseOperator` from [`packages/ui-tars/sdk/src/base/index.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/packages/ui-tars/sdk/src/base/index.ts) (lines 40-46), which provides the type signatures while leaving the concrete implementation to subclasses.

## Building a Minimal Custom Operator

Create a concrete implementation by importing the `Operator` class from `@ui-tars/sdk` and implementing the required methods:

```typescript
import { Operator, ExecuteParams, ExecuteOutput, ScreenshotOutput } from '@ui-tars/sdk';
import { Jimp } from 'jimp';

class MyOperator extends Operator {
  // Capture current UI state
  screenshot = async (): Promise<ScreenshotOutput> => {
    const img = await new Jimp({ width: 1280, height: 720, color: 0xffffffff });
    const buf = await img.getBuffer('image/png');
    return {
      base64: buf.toString('base64'),
      width: 1280,
      height: 720,
      scaleFactor: 1,
    };
  };

  // Execute parsed predictions from the model
  execute = async (params: ExecuteParams): Promise<ExecuteOutput> => {
    console.log('Executing action:', params.parsedPrediction);
    // Implement actual UI interactions (click, type, etc.) here
    return { status: StatusEnum.RUNNING };
  };
}

```

## Customizing Action Spaces with the MANUAL Field

Override the default action space by defining a static `MANUAL` field on your class. As demonstrated in [`packages/ui-tars/sdk/tests/GUIAgent.test.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/packages/ui-tars/sdk/tests/GUIAgent.test.ts) (lines 29-36), the SDK embeds these definitions into the system prompt automatically.

```typescript
class MySpecialOperator extends Operator {
  static MANUAL = {
    ACTION_SPACES: [
      `CLICK(start_box='[x1, y1, x2, y2]')`,
      `TYPE(text='your text')`,
      `END()`,
    ],
  };

  // screenshot and execute implementations as above
}

```

## Integrating Your Operator with GUIAgent

The `GUIAgent` class orchestrates the interaction between your operator and the AI model. Pass your custom operator instance via the `operator` field in the configuration:

```typescript
import { GUIAgent } from '@ui-tars/sdk';
import { MyOperator } from './my-operator';

const agent = new GUIAgent({
  model: { 
    baseURL: 'http://localhost:3000/v1', 
    apiKey: 'YOUR_KEY', 
    model: 'ui-tars' 
  },
  operator: new MyOperator(),
  onData: ({ data }) => console.log('Agent data:', data),
  onError: ({ data, error }) => console.error('Agent error:', error),
});

await agent.run('click the Submit button');

```

The `GUIAgent` calls `operator.screenshot()` once per turn, feeds the screenshot to the model, receives a prediction, then invokes `operator.execute()` with the parsed prediction data.

## Key Source Files and Architecture

| File | Purpose |
|------|---------|
| [`packages/ui-tars/sdk/src/types.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/packages/ui-tars/sdk/src/types.ts) | Abstract `Operator` class definition (lines 66-73) |
| [`packages/ui-tars/sdk/src/base/index.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/packages/ui-tars/sdk/src/base/index.ts) | `BaseOperator` base class (lines 40-46) |
| [`packages/ui-tars/sdk/src/GUIAgent.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/packages/ui-tars/sdk/src/GUIAgent.ts) | Core agent that orchestrates the operator |
| [`packages/ui-tars/sdk/tests/GUIAgent.test.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/packages/ui-tars/sdk/tests/GUIAgent.test.ts) | Test cases including custom action spaces (lines 29-36) |
| [`packages/ui-tars/sdk/src/context/useContext.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/packages/ui-tars/sdk/src/context/useContext.ts) | Utility to access SDK context inside `execute` |

## Summary

- **Extend** the abstract `Operator` class from `@ui-tars/sdk` to create your implementation.
- **Implement** `screenshot()` to return base64-encoded images with dimensions and scale factors.
- **Implement** `execute()` to handle `ExecuteParams` and return `ExecuteOutput` with appropriate status enums.
- **Define** a static `MANUAL` field to customize available action spaces for the model.
- **Inject** your operator instance into `GUIAgent` via the configuration object.

## Frequently Asked Questions

### What is the purpose of the Operator class in UI-TARS?

The `Operator` class serves as the bridge between the AI model and your specific UI environment. According to the SDK source code in [`packages/ui-tars/sdk/src/types.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/packages/ui-tars/sdk/src/types.ts), it abstractly defines how the agent captures screenshots and executes predicted actions, allowing you to implement platform-specific logic for any GUI framework.

### Do I need to implement both screenshot and execute methods?

Yes. The UI-TARS SDK requires both methods as defined in the abstract class at [`packages/ui-tars/sdk/src/types.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/packages/ui-tars/sdk/src/types.ts) (lines 66-73). The `screenshot()` method enables the model to perceive the current UI state, while `execute()` translates the model's predictions into actual UI interactions. Omitting either method will result in a compilation error when extending the abstract class.

### Can I customize which actions the model can use?

Yes. Define a static `MANUAL` field containing an `ACTION_SPACES` array on your operator class. As shown in [`packages/ui-tars/sdk/tests/GUIAgent.test.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/packages/ui-tars/sdk/tests/GUIAgent.test.ts) (lines 29-36), the SDK automatically injects these action definitions into the system prompt sent to the model, restricting it to only the actions your implementation supports.

### How do I access SDK context inside my operator?

Import and call `useContext()` from [`packages/ui-tars/sdk/src/context/useContext.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/packages/ui-tars/sdk/src/context/useContext.ts) within your `execute` method. This utility provides access to the SDK's internal context, enabling logging, configuration access, and state management during action execution.