UI-TARS Desktop Internal Action Space Types and Agent Loop Processing Explained

UI-TARS Desktop uses four reserved internal action types—call_user, max_loop, error_env, and finished—to control agent loop flow, while the GUIAgent.run() method orchestrates screenshot capture, LLM inference, and execution cycling in packages/ui-tars/sdk/src/GUIAgent.ts.

The bytedance/UI-TARS-desktop repository implements a GUI-agent architecture where the LLM generates predictions that get parsed into executable actions. Understanding the distinction between user-facing GUI commands and internal control-flow signals is critical for customizing agent behavior or debugging termination conditions.

Internal Action Space Types

The SDK defines a strict enum called INTERNAL_ACTION_SPACES_ENUM in packages/ui-tars/sdk/src/constants.ts. These values act as reserved placeholders in the system prompt template (DEFAULT_ACTION_SPACES), allowing the LLM to emit control signals without executing actual GUI operations.

export enum INTERNAL_ACTION_SPACES_ENUM {
  CALL_USER = 'call_user',
  MAX_LOOP  = 'max_loop',
  ERROR_ENV = 'error_env',
  FINISHED  = 'finished',
}

CALL_USER: Request Human Intervention

When the LLM determines a task is unsolvable or requires clarification, it emits call_user. The agent loop sets data.status = StatusEnum.CALL_USER and immediately breaks out of the prediction-handling loop, preserving the conversation state for human review.

MAX_LOOP: Iteration Limit Reached

If the internal loop counter exceeds the configured maxLoopCnt, the agent translates this into ErrorStatusEnum.REACH_MAXLOOP_ERROR. This triggers data.status = StatusEnum.ERROR and aborts the current cycle, preventing infinite execution loops.

ERROR_ENV: Environment Failure

This action signals infrastructure problems such as screenshot capture failures or display connectivity issues. The runtime maps this to ErrorStatusEnum.ENVIRONMENT_ERROR, sets the status to ERROR, and terminates the loop to prevent cascading failures.

FINISHED: Task Completion

When the LLM confirms the task objective is met, it returns finished. The agent sets data.status = StatusEnum.END and gracefully exits the loop, returning control to the caller with a success state.

Agent Loop Processing in GUIAgent.run()

The core execution logic resides in packages/ui-tars/sdk/src/GUIAgent.ts (approximately lines 310-340). The run() method implements a robust cycle that processes predictions, delegates execution to an Operator, and handles internal action resolution at specific lifecycle points.

Step 1: Internal Action Detection (Pre-Execution)

Before invoking GUI operations, the loop checks for environment or configuration errors:

if (actionType === INTERNAL_ACTION_SPACES_ENUM.ERROR_ENV) {
  Object.assign(data, {
    status: StatusEnum.ERROR,
    error: this.guiAgentErrorParser(ErrorStatusEnum.ENVIRONMENT_ERROR),
  });
  break;
} else if (actionType === INTERNAL_ACTION_SPACES_ENUM.MAX_LOOP) {
  Object.assign(data, {
    status: StatusEnum.ERROR,
    error: this.guiAgentErrorParser(ErrorStatusEnum.REACH_MAXLOOP_ERROR),
  });
  break;
}

These checks occur immediately after parsing the prediction, ensuring the agent aborts before attempting potentially unsafe GUI executions in corrupted environments.

Step 2: GUI Action Execution via Operator

For valid GUI actions, the loop delegates to the Operator abstraction using async retry logic:

if (!signal?.aborted && !this.isStopped) {
  const executeOutput = await asyncRetry(
    () => operator.execute({
      prediction,
      parsedPrediction,
      screenWidth: width,
      screenHeight: height,
      scaleFactor: snapshot.scaleFactor,
      factors: this.model.factors,
    }),
    { 
      retries: retry?.execute?.maxRetries ?? 0, 
      minTimeout: 5000, 
      onRetry: retry?.execute?.onRetry 
    }
  ).catch(e => {
    logger.error('[GUIAgent] execute error', e);
    Object.assign(data, {
      status: StatusEnum.ERROR,
      error: this.guiAgentErrorParser(ErrorStatusEnum.EXECUTE_RETRY_ERROR, e),
    });
  });
}

The operator.execute() method receives screen dimensions, scale factors, and parsed prediction data to perform actual cursor movements, keystrokes, or other system interactions.

Step 3: Post-Execution Termination Checks

After successful execution (or if execution was skipped), the loop evaluates terminal internal actions:

if (actionType === INTERNAL_ACTION_SPACES_ENUM.CALL_USER) {
  data.status = StatusEnum.CALL_USER;
  break;
} else if (actionType === INTERNAL_ACTION_SPACES_ENUM.FINISHED) {
  data.status = StatusEnum.END;
  break;
}

This ordering allows GUI actions to complete before checking for completion signals, ensuring the final state is captured before termination.

Step 4: Configurable Loop Throttling

Between iterations, the agent optionally pauses to respect resource constraints or API rate limits:

if (this.config.loopIntervalInMs && this.config.loopIntervalInMs > 0) {
  logger.info(`[GUIAgent] sleep for ${this.config.loopIntervalInMs}ms before next loop`);
  await sleep(this.config.loopIntervalInMs);
}

The complete high-level flow follows this sequence: screenshot capture → system prompt construction → LLM inference → prediction parsing → internal error handling → operator execution → termination checks → throttling sleep.

Code Examples

Extending Action Spaces with Custom Operators

You can inject custom actions into the prompt template while preserving internal action handling:

import { GUIAgent, INTERNAL_ACTION_SPACES_ENUM, DEFAULT_ACTION_SPACES } from '@ui-tars/sdk';
import { Operator, StatusEnum } from '@ui-tars/sdk/src/types';

const myActionSpaces = `
  ${DEFAULT_ACTION_SPACES}
  customAction(param='<value>')
`;

class MyOperator extends Operator {
  async execute({ parsedPrediction }) {
    if (parsedPrediction.action_type === 'customAction') {
      // Custom logic here
      return { status: StatusEnum.END };
    }
    return super.execute(arguments[0]);
  }
}

const agent = new GUIAgent({
  operator: new MyOperator(),
  systemPrompt: SYSTEM_PROMPT_TEMPLATE.replace('{{action_spaces_holder}}', myActionSpaces),
  onData: ({ data }) => console.log('agent data →', data),
});
await agent.run();

Note: Even with custom actions, the LLM may still emit internal actions (call_user, finished, etc.), which GUIAgent.run() processes automatically according to the enum definitions.

Detecting Max-Loop Termination

Configure iteration limits and handle the resulting error state:

const agent = new GUIAgent({
  operator,
  maxLoopCnt: 5,
  onData: ({ data }) => {
    if (data.status === StatusEnum.ERROR && 
        data.error?.type === ErrorStatusEnum.REACH_MAXLOOP_ERROR) {
      console.error('Agent exceeded maximum iterations');
    }
  },
});

await agent.run();
// If iterations exceed 5, the agent emits INTERNAL_ACTION_SPACES_ENUM.MAX_LOOP
// and sets final status to StatusEnum.ERROR

Summary

• Four reserved internal actions (CALL_USER, MAX_LOOP, ERROR_ENV, FINISHED) defined in packages/ui-tars/sdk/src/constants.ts control agent loop termination and error handling.
• Agent loop processing occurs in GUIAgent.run() at packages/ui-tars/sdk/src/GUIAgent.ts, which orchestrates screenshot capture, LLM inference, and execution cycling.
• Pre-execution checks handle ERROR_ENV and MAX_LOOP before GUI operations begin, while post-execution checks process CALL_USER and FINISHED after actions complete.
• Operator abstraction delegates actual GUI execution, allowing custom implementations while the core loop manages internal action semantics and retry logic.
• Configurable throttling via loopIntervalInMs provides rate limiting between iterations.

Frequently Asked Questions

What happens when the agent encounters a MAX_LOOP internal action?

When INTERNAL_ACTION_SPACES_ENUM.MAX_LOOP is detected, the agent sets data.status to StatusEnum.ERROR and maps the condition to ErrorStatusEnum.REACH_MAXLOOP_ERROR using guiAgentErrorParser(). The loop immediately breaks, terminating the run and returning control to the caller with an error state indicating the iteration limit was exceeded.

How does UI-TARS Desktop distinguish between internal actions and GUI operations?

Internal actions are enumerated in INTERNAL_ACTION_SPACES_ENUM and processed via conditional logic inside the GUIAgent.run() loop before and after operator execution. Normal GUI actions (click, type, scroll) bypass these internal checks and are passed directly to the operator.execute() method, which handles domain-specific implementations like BrowserOperator or NutJsOperator.

Where is the agent loop termination logic implemented?

The termination logic resides in packages/ui-tars/sdk/src/GUIAgent.ts within the run() method, specifically around lines 310-340. This section contains the for...of loop over parsedPredictions that evaluates actionType against INTERNAL_ACTION_SPACES_ENUM values and breaks execution upon encountering terminal states (FINISHED, CALL_USER) or error states (ERROR_ENV, MAX_LOOP).

Can developers add custom internal action types?

While you cannot modify the INTERNAL_ACTION_SPACES_ENUM without forking the SDK, you can extend DEFAULT_ACTION_SPACES in the system prompt template to include custom action signatures. However, these custom actions execute through the standard operator.execute() pathway rather than the internal action handler. To achieve internal-action-like behavior (immediate loop termination), you would need to subclass GUIAgent and override the prediction processing logic in the run() method.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}