# UI-TARS Desktop Internal Action Space Types and Agent Loop Processing Explained

> Discover UI-TARS Desktop's internal action types call user, max loop, error env, and finished. Learn how GUIAgent run() orchestrates screenshot capture, LLM inference, and execution cycling.

- Repository: [Bytedance Inc./UI-TARS-desktop](https://github.com/bytedance/UI-TARS-desktop)
- Tags: internals
- Published: 2026-05-10

---

**UI-TARS Desktop uses four reserved internal action types—`call_user`, `max_loop`, `error_env`, and `finished`—to control agent loop flow, while the `GUIAgent.run()` method orchestrates screenshot capture, LLM inference, and execution cycling in [`packages/ui-tars/sdk/src/GUIAgent.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/packages/ui-tars/sdk/src/GUIAgent.ts).**

The bytedance/UI-TARS-desktop repository implements a GUI-agent architecture where the LLM generates predictions that get parsed into executable actions. Understanding the distinction between user-facing GUI commands and internal control-flow signals is critical for customizing agent behavior or debugging termination conditions.

## Internal Action Space Types

The SDK defines a strict enum called `INTERNAL_ACTION_SPACES_ENUM` in [`packages/ui-tars/sdk/src/constants.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/packages/ui-tars/sdk/src/constants.ts). These values act as reserved placeholders in the system prompt template (`DEFAULT_ACTION_SPACES`), allowing the LLM to emit control signals without executing actual GUI operations.

```ts
export enum INTERNAL_ACTION_SPACES_ENUM {
  CALL_USER = 'call_user',
  MAX_LOOP  = 'max_loop',
  ERROR_ENV = 'error_env',
  FINISHED  = 'finished',
}

```

### CALL_USER: Request Human Intervention

When the LLM determines a task is unsolvable or requires clarification, it emits `call_user`. The agent loop sets `data.status = StatusEnum.CALL_USER` and immediately breaks out of the prediction-handling loop, preserving the conversation state for human review.

### MAX_LOOP: Iteration Limit Reached

If the internal loop counter exceeds the configured `maxLoopCnt`, the agent translates this into `ErrorStatusEnum.REACH_MAXLOOP_ERROR`. This triggers `data.status = StatusEnum.ERROR` and aborts the current cycle, preventing infinite execution loops.

### ERROR_ENV: Environment Failure

This action signals infrastructure problems such as screenshot capture failures or display connectivity issues. The runtime maps this to `ErrorStatusEnum.ENVIRONMENT_ERROR`, sets the status to `ERROR`, and terminates the loop to prevent cascading failures.

### FINISHED: Task Completion

When the LLM confirms the task objective is met, it returns `finished`. The agent sets `data.status = StatusEnum.END` and gracefully exits the loop, returning control to the caller with a success state.

## Agent Loop Processing in GUIAgent.run()

The core execution logic resides in [`packages/ui-tars/sdk/src/GUIAgent.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/packages/ui-tars/sdk/src/GUIAgent.ts) (approximately lines 310-340). The `run()` method implements a robust cycle that processes predictions, delegates execution to an **Operator**, and handles internal action resolution at specific lifecycle points.

### Step 1: Internal Action Detection (Pre-Execution)

Before invoking GUI operations, the loop checks for environment or configuration errors:

```ts
if (actionType === INTERNAL_ACTION_SPACES_ENUM.ERROR_ENV) {
  Object.assign(data, {
    status: StatusEnum.ERROR,
    error: this.guiAgentErrorParser(ErrorStatusEnum.ENVIRONMENT_ERROR),
  });
  break;
} else if (actionType === INTERNAL_ACTION_SPACES_ENUM.MAX_LOOP) {
  Object.assign(data, {
    status: StatusEnum.ERROR,
    error: this.guiAgentErrorParser(ErrorStatusEnum.REACH_MAXLOOP_ERROR),
  });
  break;
}

```

These checks occur immediately after parsing the prediction, ensuring the agent aborts before attempting potentially unsafe GUI executions in corrupted environments.

### Step 2: GUI Action Execution via Operator

For valid GUI actions, the loop delegates to the **Operator** abstraction using async retry logic:

```ts
if (!signal?.aborted && !this.isStopped) {
  const executeOutput = await asyncRetry(
    () => operator.execute({
      prediction,
      parsedPrediction,
      screenWidth: width,
      screenHeight: height,
      scaleFactor: snapshot.scaleFactor,
      factors: this.model.factors,
    }),
    { 
      retries: retry?.execute?.maxRetries ?? 0, 
      minTimeout: 5000, 
      onRetry: retry?.execute?.onRetry 
    }
  ).catch(e => {
    logger.error('[GUIAgent] execute error', e);
    Object.assign(data, {
      status: StatusEnum.ERROR,
      error: this.guiAgentErrorParser(ErrorStatusEnum.EXECUTE_RETRY_ERROR, e),
    });
  });
}

```

The `operator.execute()` method receives screen dimensions, scale factors, and parsed prediction data to perform actual cursor movements, keystrokes, or other system interactions.

### Step 3: Post-Execution Termination Checks

After successful execution (or if execution was skipped), the loop evaluates terminal internal actions:

```ts
if (actionType === INTERNAL_ACTION_SPACES_ENUM.CALL_USER) {
  data.status = StatusEnum.CALL_USER;
  break;
} else if (actionType === INTERNAL_ACTION_SPACES_ENUM.FINISHED) {
  data.status = StatusEnum.END;
  break;
}

```

This ordering allows GUI actions to complete before checking for completion signals, ensuring the final state is captured before termination.

### Step 4: Configurable Loop Throttling

Between iterations, the agent optionally pauses to respect resource constraints or API rate limits:

```ts
if (this.config.loopIntervalInMs && this.config.loopIntervalInMs > 0) {
  logger.info(`[GUIAgent] sleep for ${this.config.loopIntervalInMs}ms before next loop`);
  await sleep(this.config.loopIntervalInMs);
}

```

The complete high-level flow follows this sequence: screenshot capture → system prompt construction → LLM inference → prediction parsing → internal error handling → operator execution → termination checks → throttling sleep.

## Code Examples

### Extending Action Spaces with Custom Operators

You can inject custom actions into the prompt template while preserving internal action handling:

```ts
import { GUIAgent, INTERNAL_ACTION_SPACES_ENUM, DEFAULT_ACTION_SPACES } from '@ui-tars/sdk';
import { Operator, StatusEnum } from '@ui-tars/sdk/src/types';

const myActionSpaces = `
  ${DEFAULT_ACTION_SPACES}
  customAction(param='<value>')
`;

class MyOperator extends Operator {
  async execute({ parsedPrediction }) {
    if (parsedPrediction.action_type === 'customAction') {
      // Custom logic here
      return { status: StatusEnum.END };
    }
    return super.execute(arguments[0]);
  }
}

const agent = new GUIAgent({
  operator: new MyOperator(),
  systemPrompt: SYSTEM_PROMPT_TEMPLATE.replace('{{action_spaces_holder}}', myActionSpaces),
  onData: ({ data }) => console.log('agent data →', data),
});
await agent.run();

```

**Note:** Even with custom actions, the LLM may still emit internal actions (`call_user`, `finished`, etc.), which `GUIAgent.run()` processes automatically according to the enum definitions.

### Detecting Max-Loop Termination

Configure iteration limits and handle the resulting error state:

```ts
const agent = new GUIAgent({
  operator,
  maxLoopCnt: 5,
  onData: ({ data }) => {
    if (data.status === StatusEnum.ERROR && 
        data.error?.type === ErrorStatusEnum.REACH_MAXLOOP_ERROR) {
      console.error('Agent exceeded maximum iterations');
    }
  },
});

await agent.run();
// If iterations exceed 5, the agent emits INTERNAL_ACTION_SPACES_ENUM.MAX_LOOP
// and sets final status to StatusEnum.ERROR

```

## Summary

- **Four reserved internal actions** (`CALL_USER`, `MAX_LOOP`, `ERROR_ENV`, `FINISHED`) defined in [`packages/ui-tars/sdk/src/constants.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/packages/ui-tars/sdk/src/constants.ts) control agent loop termination and error handling.
- **Agent loop processing** occurs in `GUIAgent.run()` at [`packages/ui-tars/sdk/src/GUIAgent.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/packages/ui-tars/sdk/src/GUIAgent.ts), which orchestrates screenshot capture, LLM inference, and execution cycling.
- **Pre-execution checks** handle `ERROR_ENV` and `MAX_LOOP` before GUI operations begin, while **post-execution checks** process `CALL_USER` and `FINISHED` after actions complete.
- **Operator abstraction** delegates actual GUI execution, allowing custom implementations while the core loop manages internal action semantics and retry logic.
- **Configurable throttling** via `loopIntervalInMs` provides rate limiting between iterations.

## Frequently Asked Questions

### What happens when the agent encounters a MAX_LOOP internal action?

When `INTERNAL_ACTION_SPACES_ENUM.MAX_LOOP` is detected, the agent sets `data.status` to `StatusEnum.ERROR` and maps the condition to `ErrorStatusEnum.REACH_MAXLOOP_ERROR` using `guiAgentErrorParser()`. The loop immediately breaks, terminating the run and returning control to the caller with an error state indicating the iteration limit was exceeded.

### How does UI-TARS Desktop distinguish between internal actions and GUI operations?

Internal actions are enumerated in `INTERNAL_ACTION_SPACES_ENUM` and processed via conditional logic inside the `GUIAgent.run()` loop before and after operator execution. Normal GUI actions (click, type, scroll) bypass these internal checks and are passed directly to the `operator.execute()` method, which handles domain-specific implementations like `BrowserOperator` or `NutJsOperator`.

### Where is the agent loop termination logic implemented?

The termination logic resides in [`packages/ui-tars/sdk/src/GUIAgent.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/packages/ui-tars/sdk/src/GUIAgent.ts) within the `run()` method, specifically around lines 310-340. This section contains the `for...of` loop over `parsedPredictions` that evaluates `actionType` against `INTERNAL_ACTION_SPACES_ENUM` values and breaks execution upon encountering terminal states (`FINISHED`, `CALL_USER`) or error states (`ERROR_ENV`, `MAX_LOOP`).

### Can developers add custom internal action types?

While you cannot modify the `INTERNAL_ACTION_SPACES_ENUM` without forking the SDK, you can extend `DEFAULT_ACTION_SPACES` in the system prompt template to include custom action signatures. However, these custom actions execute through the standard `operator.execute()` pathway rather than the internal action handler. To achieve internal-action-like behavior (immediate loop termination), you would need to subclass `GUIAgent` and override the prediction processing logic in the `run()` method.