How to Implement Custom System Prompts for Specific UI-TARS Use Cases

You can implement custom system prompts in UI-TARS by passing a systemPrompt string to the GUIAgent constructor, defining a custom Operator class with MANUAL.ACTION_SPACES, or modifying the SYSTEM_PROMPT_TEMPLATE constant.

The UI-TARS SDK from the bytedance/UI-TARS-desktop repository provides a flexible agent architecture that allows you to customize how the LLM interprets screenshots and actions. Implementing custom system prompts enables domain-specific workflows, specialized vocabulary, and constrained action spaces for your GUI automation tasks.

Direct System Prompt Override

The simplest method to implement a custom system prompt is passing the systemPrompt field directly to the GUIAgent configuration. According to the source code in packages/ui-tars/sdk/src/GUIAgent.ts (lines 63-66), the constructor sets this.systemPrompt = config.systemPrompt || this.buildSystemPrompt();, which means any string you provide takes precedence over the default template.

This approach is ideal when you need complete control over the LLM instructions, output format, and action definitions.

import { GUIAgent } from '@ui-tars/sdk';
import { NutJSOperator } from '@ui-tars/operator-nut-js';

const customPrompt = `
You are a specialised finance‑assistant GUI agent.
Your task is to automate spreadsheet operations in Excel.

## Output Format

\`\`\`
Thought: ...
Action: ...
\`\`\`

## Action Space

click(start_box='[x1, y1, x2, y2]')
type(content='')               # type into the active cell

hotkey(key='Ctrl+S')           # save the workbook

finished()
call_user()
`;

const agent = new GUIAgent({
  model: { baseURL: '<your‑endpoint>', apiKey: '<key>', model: 'gpt-4o' },
  operator: new NutJSOperator(),
  systemPrompt: customPrompt,   // ← custom system prompt injected here
});
await agent.run('Open the financial report and paste the latest numbers.');

The customPrompt string completely replaces the default prompt, allowing you to embed domain-specific instructions, custom action definitions, or contextual hints.

Custom Operator Action Spaces

For reusable, domain-specific agents, define a custom operator that exposes its own MANUAL.ACTION_SPACES. The SDK automatically injects these action definitions into the system prompt template at runtime.

In packages/ui-tars/sdk/src/GUIAgent.ts (lines 15-25), the buildSystemPrompt() method replaces the {{action_spaces_holder}} placeholder with the operator's static MANUAL.ACTION_SPACES property. If the operator provides no custom actions, the SDK falls back to the generic SYSTEM_PROMPT constant defined in packages/ui-tars/sdk/src/constants.ts.

import { Operator } from '@ui-tars/sdk/core';
import { GUIAgent } from '@ui-tars/sdk';
import { Jimp } from 'jimp';

export class SpreadsheetOperator extends Operator {
  // The static MANUAL field is read by GUIAgent.buildSystemPrompt()
  static MANUAL = {
    ACTION_SPACES: [
      "click(start_box='[x1, y1, x2, y2]')",
      "type(content='') # type into the active cell",

      "hotkey(key='Ctrl+S') # save workbook",

      "finished()",
      "call_user()",
    ],
  };

  async screenshot() {
    // Capture screen as base64 (implementation omitted)
    return { base64: '<base64>', scaleFactor: 1 };
  }

  async execute(params) {
    // Translate prediction into real OS actions (implementation omitted)
    return { status: 'running' };
  }
}

// The SDK will automatically splice SpreadsheetOperator.MANUAL.ACTION_SPACES
// into the SYSTEM_PROMPT_TEMPLATE.
const agent = new GUIAgent({
  model: { baseURL: '<url>', apiKey: '<key>', model: 'gpt-4o-mini' },
  operator: new SpreadsheetOperator(),
});
await agent.run('Insert a chart showing quarterly revenue.');

No explicit systemPrompt is required when using this method. Adding, removing, or re-ordering entries in ACTION_SPACES instantly updates the model's available actions without modifying the base template.

Global Template Customization

For baseline changes that apply to every agent instance in your application, modify the SYSTEM_PROMPT_TEMPLATE constant in packages/ui-tars/sdk/src/constants.ts (lines 9-29). This template uses the {{action_spaces_holder}} placeholder to indicate where action spaces are injected during buildSystemPrompt().

// packages/ui-tars/sdk/src/constants.ts
export const SYSTEM_PROMPT_TEMPLATE = `You are a GUI agent designed for multi‑step workflow automation.
{{action_spaces_holder}}

## Note

- Always summarize the next step in the Thought section.

## User Instruction

`;

After rebuilding the SDK (e.g., npm run build), all agents that rely on the default template will use the updated wording. This method is best suited for organization-wide defaults rather than use-case-specific customization.

How Prompt Resolution Works

Understanding the internal prompt resolution flow helps debug custom implementations:

1. Agent instantiation – The GUIAgent constructor checks config.systemPrompt. If absent, it invokes buildSystemPrompt() (lines 63-66 in GUIAgent.ts).
2. Template construction – buildSystemPrompt() retrieves MANUAL.ACTION_SPACES from the provided operator class. It substitutes the {{action_spaces_holder}} placeholder in SYSTEM_PROMPT_TEMPLATE with these action definitions (lines 15-25).
3. Runtime injection – During agent.run(), the resolved systemPrompt is added to the request payload as data.systemPrompt (lines 82-86 in GUIAgent.ts).
4. Model processing – The LLM receives the complete system prompt string to guide its reasoning and output format according to your custom actions or instructions.

Summary

• Direct override provides immediate, complete control via the systemPrompt config field in GUIAgent.
• Custom operators enable reusable, domain-specific action spaces through the static MANUAL.ACTION_SPACES property.
• Global templates allow baseline modifications via SYSTEM_PROMPT_TEMPLATE in constants.ts.
• The SDK resolves prompts at construction time, falling back to buildSystemPrompt() when no explicit custom prompt is provided.

Frequently Asked Questions

What is the difference between SYSTEM_PROMPT_TEMPLATE and SYSTEM_PROMPT?

SYSTEM_PROMPT_TEMPLATE (defined in packages/ui-tars/sdk/src/constants.ts) contains the placeholder {{action_spaces_holder}} and serves as the base structure for dynamically generated prompts. SYSTEM_PROMPT is the static fallback used when an operator provides no custom MANUAL.ACTION_SPACES. The template offers flexibility, while the static constant provides a generic default.

Can I combine custom system prompts with custom operators?

Yes, but the explicit systemPrompt string in the GUIAgent config takes precedence. If you pass both a systemPrompt and a custom operator with MANUAL.ACTION_SPACES, the SDK uses your explicit string and ignores the operator's action space injection. To utilize operator-defined actions, omit the systemPrompt field from the configuration.

Where does the system prompt get injected into the LLM request?

The system prompt is injected during the run() method execution in packages/ui-tars/sdk/src/GUIAgent.ts (lines 82-86), where data.systemPrompt is added to the payload sent to the model endpoint. This occurs after the constructor has resolved the final prompt string via either direct assignment or template building.

How do I add new GUI actions without modifying the source code?

Define a custom operator class extending Operator with a static MANUAL.ACTION_SPACES array containing your new action definitions. Pass this operator to the GUIAgent constructor. The SDK automatically splices these actions into the system prompt template without requiring changes to constants.ts or the core SDK files.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}