How Screenshot Retry Logic Works in UI-TARS Desktop: Configuration and Customization Guide

UI-TARS Desktop wraps the operator's screenshot() method with async-retry to automatically handle transient failures, allowing developers to customize retry attempts and callbacks via the RetryConfig interface in GUIAgentConfig.

UI-TARS Desktop captures a screenshot before every model inference to understand the current UI state. The screenshot retry logic, implemented in the GUIAgent class within the bytedance/UI-TARS-desktop repository, uses the async-retry library to make the agent resilient against temporary glitches or flaky ADB connections.

How Screenshot Retry Logic Works in UI-TARS Desktop

The core retry mechanism resides in packages/ui-tars/sdk/src/GUIAgent.ts. Before each model inference, the agent invokes the operator's screenshot method through an asyncRetry wrapper:

// packages/ui-tars/sdk/src/GUIAgent.ts (lines 85-89)
const snapshot = await asyncRetry(
  () => operator.screenshot(),
  {
    retries: retry?.screenshot?.maxRetries ?? 0,
    minTimeout: 5000,
    onRetry: retry?.screenshot?.onRetry,
  },
);

Default behavior sets maxRetries to 0 when no configuration is provided, meaning the screenshot is attempted only once. If this single attempt fails, the agent aborts and reports ErrorStatusEnum.SCREENSHOT_RETRY_ERROR. The retry logic specifically targets only the operator.screenshot() function—image processing, model invocation, and action execution remain outside this retry scope.

The minTimeout is fixed at 5000 milliseconds (5 seconds) to prevent overwhelming the device with rapid successive attempts.

Customizing Screenshot Retry Configuration

Developers inject custom retry policies through the GUIAgentConfig object passed to the GUIAgent constructor.

Understanding the RetryConfig Interface

The retry configuration is defined in packages/ui-tars/sdk/src/types.ts:

export interface GUIAgentConfig<TOperator> {
  // ... other properties
  retry?: {
    model?: RetryConfig;
    screenshot?: RetryConfig;
    execute?: RetryConfig;
  };
}

export interface RetryConfig {
  maxRetries: number;           // Additional attempts after first failure
  onRetry?: (error: Error, attempt: number) => void;  // Failure callback
}

Setting maxRetries: 3 results in four total attempts (initial plus three retries). The optional onRetry callback executes after each failure, enabling logging or adaptive backoff strategies.

Implementing Custom Retry Policies

Example: Enable three extra screenshot attempts with logging:

import { GUIAgent } from '@ui-tars/sdk';
import { MyOperator } from './my-operator';

const agent = new GUIAgent({
  operator: new MyOperator(),
  model: myModel,
  retry: {
    screenshot: {
      maxRetries: 3,
      onRetry: (err, attempt) => {
        console.warn(
          `[GUIAgent] screenshot retry #${attempt} failed:`,
          err.message,
        );
      },
    },
  },
});

Example: Configure retries across all stages:

const agent = new GUIAgent({
  operator: new MyOperator(),
  model: myModel,
  retry: {
    model: { maxRetries: 2 },
    screenshot: { maxRetries: 3 },
    execute: { maxRetries: 1 },
  },
});

Error Handling and Loop Interaction

When screenshot retries are exhausted, the agent increments snapshotErrCnt and pauses for one second (await sleep(1000)) before attempting the next cycle. This continues until MAX_SNAPSHOT_ERR_CNT (defined in packages/ui-tars/shared/constants.ts) is reached, at which point the agent terminates with ErrorStatusEnum.SCREENSHOT_RETRY_ERROR.

If the screenshot succeeds within the allotted retries, the captured image is stored in the conversations array and emitted via the onData callback, allowing the model inference loop to proceed normally.

Summary

• Location: The retry logic lives in packages/ui-tars/sdk/src/GUIAgent.ts, wrapping operator.screenshot() with async-retry.
• Defaults: Zero retries by default (maxRetries: 0) with a fixed 5-second minimum timeout between attempts.
• Configuration: Customize via RetryConfig in GUIAgentConfig, setting maxRetries and an optional onRetry callback.
• Limits: Persistent failures increment snapshotErrCnt until MAX_SNAPSHOT_ERR_CNT triggers agent termination.

Frequently Asked Questions

How do I completely disable screenshot retries?

Omit the retry.screenshot configuration or explicitly set maxRetries: 0. Since the default value is 0, the agent attempts the screenshot exactly once and immediately fails on any error.

Can I change the delay between retry attempts?

The minTimeout parameter is hardcoded to 5000 milliseconds in the asyncRetry call within GUIAgent.ts. To modify this value, you must patch the source code or fork the repository, as the SDK deliberately maintains this fixed interval to prevent device hammering.

What happens if all screenshot retries fail?

When all retry attempts exhaust, the agent increments the internal snapshotErrCnt counter, pauses for one second, and attempts to continue. If failures persist until reaching MAX_SNAPSHOT_ERR_CNT (typically defined in packages/ui-tars/shared/constants.ts), the agent aborts with ErrorStatusEnum.SCREENSHOT_RETRY_ERROR.

Does the screenshot retry logic apply to the model API calls?

No. The retry configuration in GUIAgentConfig supports separate policies for model, screenshot, and execute stages. The screenshot retry only wraps the operator's screenshot() method. Model inference retries require separate configuration under retry.model.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}