UI-TARS SDK Retry Mechanism: How to Configure Retries for Screenshots, Models, and Actions

The UI-TARS SDK implements a configurable retry mechanism using the async-retry library (v1.3.3) for screenshot capture, model inference, and action execution, defaulting to zero retries but allowing granular per-operation configuration via the RetryConfig interface.

The retry system in the bytedance/UI-TARS-desktop repository provides resilience for the three critical operations within the GUIAgent class. By default, the SDK performs no retries, but you can enable automatic re-attempts with custom callbacks and timeouts for specific failure scenarios.

How the UI-TARS SDK Retry Mechanism Works

The retry logic is centralized in the GUIAgent.run method within packages/ui-tars/sdk/src/GUIAgent.ts. The SDK wraps three distinct operations with the async-retry utility to handle transient failures in GUI automation workflows.

Retry Operations and Default Behavior

The SDK applies retry logic to three specific phases of the agent execution cycle:

Operation	Location in SDK	Default Max Retries	Min Timeout
Screenshot	GUIAgent.run lines 85-89	0	5000ms
Model invocation	GUIAgent.run lines 96-100	0	30000ms
Action execution	GUIAgent.run lines 122-126	0	5000ms

Each operation defaults to maxRetries: 0, meaning the SDK attempts each operation exactly once unless explicitly configured otherwise.

Underlying Implementation with async-retry

The mechanism relies on the async-retry npm package declared in packages/ui-tars/sdk/package.json. Inside GUIAgent.run, each operation calls asyncRetry with operation-specific parameters:

// Screenshot retry logic (GUIAgent.ts lines 85-89)
await asyncRetry(() => operator.screenshot(), {
  retries: retry?.screenshot?.maxRetries ?? 0,
  minTimeout: 5000,
  onRetry: retry?.screenshot?.onRetry,
});

// Model invocation retry logic (GUIAgent.ts lines 96-100)
await asyncRetry(async (bail) => { /* model call */ }, {
  retries: retry?.model?.maxRetries ?? 0,
  minTimeout: 30_000,
  onRetry: retry?.model?.onRetry,
});

// Action execution retry logic (GUIAgent.ts lines 122-126)
await asyncRetry(() => operator.execute({ /* action */ }), {
  retries: retry?.execute?.maxRetries ?? 0,
  minTimeout: 5000,
  onRetry: retry?.execute?.onRetry,
});

The bail function available in the model invocation wrapper allows immediate termination of retries for non-recoverable errors.

Configuration Architecture

The retry system uses a strongly-typed configuration structure defined in packages/ui-tars/sdk/src/types.ts.

RetryConfig Interface

The SDK exposes a RetryConfig interface at lines 81-84:

interface RetryConfig {
  maxRetries?: number;
  onRetry?: (error: Error, attempt: number) => void | Promise<void>;
}

• maxRetries: Specifies the number of additional attempts beyond the initial call (e.g., maxRetries: 2 allows 3 total attempts).
• onRetry: Optional callback invoked after each failed attempt, receiving the error and attempt number (1-indexed).

GUIAgentConfig Integration

The top-level configuration accepts an optional retry field containing separate configs for each operation (lines 96-103):

interface GUIAgentConfig {
  // ... other config options
  retry?: {
    model?: RetryConfig;
    screenshot?: RetryConfig;
    execute?: RetryConfig;
  };
}

This structure allows independent retry strategies for each phase of the automation loop.

Configuring Retries in Practice

When instantiating GUIAgent, pass the retry configuration object to enable resilience for specific operations.

Basic Configuration: Screenshot Retries

Enable retries only for screenshot operations when dealing with unstable screen capture:

import { GUIAgent } from '@ui-tars/sdk';

const agent = new GUIAgent({
  operator: new MyOperator(),
  model: new MyModel(),
  retry: {
    screenshot: {
      maxRetries: 2,
      onRetry: (err, attempt) => {
        console.warn(`Screenshot failed, retry #${attempt}: ${err.message}`);
      },
    },
    // model and execute remain at default (no retries)
  },
});

await agent.run('Open the settings page');

This configuration attempts screenshots up to 3 times total (initial + 2 retries) with a 5000ms minimum timeout between attempts.

Comprehensive Configuration: All Operations

Configure different retry limits for each phase of the agent loop:

const agent = new GUIAgent({
  operator: new MyOperator(),
  model: new MyModel(),
  retry: {
    screenshot: { 
      maxRetries: 1  // 2 attempts total
    },
    model: {
      maxRetries: 3,  // 4 attempts total for LLM calls
      onRetry: (e, i) => console.info(`Model retry ${i}`, e),
    },
    execute: { 
      maxRetries: 2,  // 3 attempts total for action execution
      onRetry: (err) => console.error('Action failed:', err)
    },
  },
});

The model invocation uses a 30-second timeout suitable for API latency, while screenshot and execution use 5-second timeouts.

Advanced: Custom onRetry Hooks with Exponential Backoff

Implement custom backoff strategies by returning a Promise from onRetry:

function exponentialBackoff(error: Error, attempt: number) {
  const delay = Math.pow(2, attempt) * 1000; // 1s, 2s, 4s...
  console.warn(`Waiting ${delay}ms before retry #${attempt}: ${error.message}`);
  return new Promise((resolve) => setTimeout(resolve, delay));
}

const agent = new GUIAgent({
  operator: new MyOperator(),
  model: new MyModel(),
  retry: {
    model: {
      maxRetries: 3,
      onRetry: exponentialBackoff,
    },
  },
});

Because async-retry respects the resolution timing of the onRetry callback, this custom delay logic properly throttles retry attempts.

Error Handling and Timeouts

When the retry limit exhausts, async-retry throws the final error, which the SDK catches and converts to a GUIAgentError via the guiAgentErrorParser utility. The error propagates through the agent's onError callback if configured.

Key timeout characteristics:

• Screenshot retries: 5000ms minimum between attempts
• Model retries: 30000ms minimum between attempts (accommodating LLM API latency)
• Action retries: 5000ms minimum between attempts

If omitted from the configuration, any operation defaults to immediate failure propagation without retry attempts.

Summary

• The UI-TARS SDK retry mechanism uses async-retry to handle transient failures in screenshot, model, and execution operations.
• Default behavior is zero retries (maxRetries: 0), requiring explicit opt-in for resilience.
• Configure retries via the RetryConfig interface in GUIAgentConfig, supporting distinct strategies per operation type.
• Implement custom logic using the onRetry callback, which receives the error and attempt count and supports asynchronous delays.
• Source implementation resides in packages/ui-tars/sdk/src/GUIAgent.ts with type definitions in packages/ui-tars/sdk/src/types.ts.

Frequently Asked Questions

What is the default retry behavior in the UI-TARS SDK?

By default, the SDK performs no retries. The maxRetries parameter defaults to 0 for screenshot, model invocation, and action execution operations, meaning each operation executes exactly once and fails immediately on error. This conservative default prevents unnecessary latency in production environments.

How do I configure different retry limits for screenshots versus model calls?

Pass separate RetryConfig objects within the retry field of GUIAgentConfig. The configuration supports distinct maxRetries values for screenshot, model, and execute properties. For example, set retry.screenshot.maxRetries: 2 for image capture resilience while keeping retry.model.maxRetries: 0 for strict single-shot LLM inference.

Can I implement exponential backoff with the UI-TARS SDK retry mechanism?

Yes. The onRetry callback in RetryConfig supports async functions. Return a Promise that resolves after your desired delay to implement exponential backoff, jitter, or other custom timing strategies. The underlying async-retry library waits for the onRetry Promise to resolve before attempting the next retry.

What happens when all retry attempts are exhausted?

When retry limits exhaust, the SDK throws the final error through the guiAgentErrorParser and propagates it via the onError callback of the GUIAgent. If unhandled, this terminates the current agent.run() execution. The error object contains the original failure details from the last attempt, allowing diagnostic logging or fallback logic in your application code.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}