How to Implement a Custom Operator Class for UI-TARS SDK

To implement a custom Operator class for the UI-TARS SDK, extend the abstract Operator class and implement the screenshot() method to capture UI state and the execute() method to handle parsed model predictions.

The UI-TARS SDK, part of the bytedance/UI-TARS-desktop repository, provides a flexible framework for building GUI automation agents. The Operator abstraction serves as the critical bridge between the AI model's predictions and your specific UI environment, requiring you to implement screenshot capture and action execution logic tailored to your application.

Understanding the Operator Contract

The UI-TARS SDK defines the operator interface in packages/ui-tars/sdk/src/types.ts (lines 66-73). According to the source code, every operator must satisfy two fundamental requirements:

• Screenshot capture: The screenshot() method must return a Promise<ScreenshotOutput> containing a base64-encoded image string, width, height, and screen scale factor.
• Action execution: The execute() method receives an ExecuteParams object and returns a Promise<ExecuteOutput> with a status of either StatusEnum.RUNNING or StatusEnum.END.

The abstract class extends BaseOperator from packages/ui-tars/sdk/src/base/index.ts (lines 40-46), which provides the type signatures while leaving the concrete implementation to subclasses.

Building a Minimal Custom Operator

Create a concrete implementation by importing the Operator class from @ui-tars/sdk and implementing the required methods:

import { Operator, ExecuteParams, ExecuteOutput, ScreenshotOutput } from '@ui-tars/sdk';
import { Jimp } from 'jimp';

class MyOperator extends Operator {
  // Capture current UI state
  screenshot = async (): Promise<ScreenshotOutput> => {
    const img = await new Jimp({ width: 1280, height: 720, color: 0xffffffff });
    const buf = await img.getBuffer('image/png');
    return {
      base64: buf.toString('base64'),
      width: 1280,
      height: 720,
      scaleFactor: 1,
    };
  };

  // Execute parsed predictions from the model
  execute = async (params: ExecuteParams): Promise<ExecuteOutput> => {
    console.log('Executing action:', params.parsedPrediction);
    // Implement actual UI interactions (click, type, etc.) here
    return { status: StatusEnum.RUNNING };
  };
}

Customizing Action Spaces with the MANUAL Field

Override the default action space by defining a static MANUAL field on your class. As demonstrated in packages/ui-tars/sdk/tests/GUIAgent.test.ts (lines 29-36), the SDK embeds these definitions into the system prompt automatically.

class MySpecialOperator extends Operator {
  static MANUAL = {
    ACTION_SPACES: [
      `CLICK(start_box='[x1, y1, x2, y2]')`,
      `TYPE(text='your text')`,
      `END()`,
    ],
  };

  // screenshot and execute implementations as above
}

Integrating Your Operator with GUIAgent

The GUIAgent class orchestrates the interaction between your operator and the AI model. Pass your custom operator instance via the operator field in the configuration:

import { GUIAgent } from '@ui-tars/sdk';
import { MyOperator } from './my-operator';

const agent = new GUIAgent({
  model: { 
    baseURL: 'http://localhost:3000/v1', 
    apiKey: 'YOUR_KEY', 
    model: 'ui-tars' 
  },
  operator: new MyOperator(),
  onData: ({ data }) => console.log('Agent data:', data),
  onError: ({ data, error }) => console.error('Agent error:', error),
});

await agent.run('click the Submit button');

The GUIAgent calls operator.screenshot() once per turn, feeds the screenshot to the model, receives a prediction, then invokes operator.execute() with the parsed prediction data.

Key Source Files and Architecture

File	Purpose
packages/ui-tars/sdk/src/types.ts	Abstract Operator class definition (lines 66-73)
packages/ui-tars/sdk/src/base/index.ts	BaseOperator base class (lines 40-46)
packages/ui-tars/sdk/src/GUIAgent.ts	Core agent that orchestrates the operator
packages/ui-tars/sdk/tests/GUIAgent.test.ts	Test cases including custom action spaces (lines 29-36)
packages/ui-tars/sdk/src/context/useContext.ts	Utility to access SDK context inside execute

Summary

• Extend the abstract Operator class from @ui-tars/sdk to create your implementation.
• Implement screenshot() to return base64-encoded images with dimensions and scale factors.
• Implement execute() to handle ExecuteParams and return ExecuteOutput with appropriate status enums.
• Define a static MANUAL field to customize available action spaces for the model.
• Inject your operator instance into GUIAgent via the configuration object.

Frequently Asked Questions

What is the purpose of the Operator class in UI-TARS?

The Operator class serves as the bridge between the AI model and your specific UI environment. According to the SDK source code in packages/ui-tars/sdk/src/types.ts, it abstractly defines how the agent captures screenshots and executes predicted actions, allowing you to implement platform-specific logic for any GUI framework.

Do I need to implement both screenshot and execute methods?

Yes. The UI-TARS SDK requires both methods as defined in the abstract class at packages/ui-tars/sdk/src/types.ts (lines 66-73). The screenshot() method enables the model to perceive the current UI state, while execute() translates the model's predictions into actual UI interactions. Omitting either method will result in a compilation error when extending the abstract class.

Can I customize which actions the model can use?

Yes. Define a static MANUAL field containing an ACTION_SPACES array on your operator class. As shown in packages/ui-tars/sdk/tests/GUIAgent.test.ts (lines 29-36), the SDK automatically injects these action definitions into the system prompt sent to the model, restricting it to only the actions your implementation supports.

How do I access SDK context inside my operator?

Import and call useContext() from packages/ui-tars/sdk/src/context/useContext.ts within your execute method. This utility provides access to the SDK's internal context, enabling logging, configuration access, and state management during action execution.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}