VLM Providers Available in UI-TARS Desktop and Their Impact on Action Parsing

UI-TARS-desktop supports four distinct VLM providers—Hugging Face for UI-TARS 1.0/1.5 and VolcEngine Ark for Doubao 1.5 variants—which determine the system prompt loaded at runtime, thereby controlling the action-parsing grammar that structures the agent's output.

The bytedance/UI-TARS-desktop repository implements a provider-agnostic agent architecture where the selected Vision-Language Model (VLM) provider dictates how natural language instructions translate into executable UI automation. Your provider selection triggers a cascade of configuration changes that determine both the model weights invoked and the structured action schema expected by the downstream parser.

Available VLM Providers in UI-TARS

The available providers are defined as enum values in apps/ui-tars/src/main/store/types.ts. UI-TARS-desktop currently supports four distinct configurations:

• VLMProviderV2.ui_tars_1_0 – Hugging Face for UI-TARS-1.0
• VLMProviderV2.ui_tars_1_5 – Hugging Face for UI-TARS-1.5
• VLMProviderV2.doubao_1_5 – VolcEngine Ark for Doubao-1.5-UI-TARS (15B)
• VLMProviderV2.doubao_1_5_vl – VolcEngine Ark for Doubao-1.5-thinking-vision-pro (20B)

Each enum value maps to a specific model version constant and a dedicated system prompt function that encodes the action-parsing grammar for that provider.

How VLM Providers Control Action Parsing

The VLM provider affects action parsing through a two-step resolution process implemented in apps/ui-tars/src/main/utils/agent.ts:

1. Model Version Resolution – The getModelVersion() function (lines 23-37) maps the provider enum to a concrete UITarsModelVersion constant.
2. System Prompt Selection – The getSpByModelVersion() function (lines 40-54) selects the provider-specific system prompt that defines the action grammar the LLM must follow.

The parser in multimodal/omni-tars/core/src/utils/parser.ts—specifically parseComputerContent (lines 181-200) and related functions—expects the model's output to conform to the schema described in the active system prompt. When you select a vision-language provider such as doubao_1_5_vl, the system prompt includes additional instructions for image-based tool calls, enabling the parser to recognize multimodal actions that standard providers cannot process.

Provider-to-Model Version Mapping

In apps/ui-tars/src/main/utils/agent.ts, the getModelVersion switch statement translates provider selection into concrete model versions:

• ui_tars_1_0 → UITarsModelVersion.V1_0
• ui_tars_1_5 → UITarsModelVersion.V1_5
• doubao_1_5 → UITarsModelVersion.DOUBAO_1_5_15B
• doubao_1_5_vl → UITarsModelVersion.DOUBAO_1_5_20B

This mapping ensures the agent invokes the correct endpoint weights for the selected provider.

System Prompt Selection and Action Grammar

The getSpByModelVersion function returns one of four system prompt generators based on the resolved model version:

• V1_0: getSystemPrompt() – Default text-only action grammar
• V1_5: getSystemPromptV1_5() – Enhanced UI-TARS 1.5 grammar
• DOUBAO_1_5_15B: getSystemPromptDoubao_15_15B() – VolcEngine standard actions
• DOUBAO_1_5_20B: getSystemPromptDoubao_15_20B() – VolcEngine vision-language actions with image support

The system prompt encodes permissible tool calls, mouse event formats, and vision-language capabilities. Consequently, switching providers changes the set of action formats that parseComputerContent can successfully validate and execute.

Configuring VLM Providers: Code Examples

Selecting a Provider and Loading the Correct Prompt

The following TypeScript example demonstrates how provider selection drives model version and system prompt resolution:

import { VLMProviderV2 } from '@/store/types';
import { getModelVersion, getSpByModelVersion } from '@/utils/agent';

// Select the Doubao 1.5 vision-language provider
const provider: VLMProviderV2 = VLMProviderV2.doubao_1_5_vl;

// Resolve to concrete model version: UITarsModelVersion.DOUBAO_1_5_20B
const modelVersion = getModelVersion(provider);

// Load the corresponding system prompt with action grammar
const systemPrompt = getSpByModelVersion(
  modelVersion,
  'en',          // language code
  'computer'     // operator type
);
// systemPrompt now contains Doubao-1.5-VL specific instructions

Source: getModelVersion and getSpByModelVersion are defined in apps/ui-tars/src/main/utils/agent.ts (lines 23-54).

Parsing Model Output According to Provider Grammar

The parser expects tool definitions that match the active system prompt's schema:

import { parseComputerContent } from '@/multimodal/omni-tars/core/src/utils/parser';

const rawResponse = `{
  "content": "move mouse to (400,300) and left_click",
  "tools": [{
    "type": "function",
    "function": {
      "name": "mouse_move",
      "arguments": "{\"x\":400,\"y\":300}"
    }
  },{
    "type": "function",
    "function": {
      "name":"left_click",
      "arguments":"{}"
    }
  }]
}`;

const parsed = parseComputerContent(rawResponse);
// parsed.tools contains structured actions for execution

Source: Action parsing logic resides in multimodal/omni-tars/core/src/utils/parser.ts (lines 181-200).

Runtime Provider Switching

You can reconfigure the agent at runtime by updating the provider and regenerating the system prompt:

function onProviderChange(newProvider: VLMProviderV2) {
  // Re-compute model version and prompt for new provider
  const version = getModelVersion(newProvider);
  const prompt = getSpByModelVersion(version, 'en', 'computer');
  
  // Update agent configuration
  agent.setSystemPrompt(prompt);
}

Summary

• UI-TARS-desktop supports four VLM providers via the VLMProviderV2 enum defined in apps/ui-tars/src/main/store/types.ts.
• Provider selection determines the model version through getModelVersion() in apps/ui-tars/src/main/utils/agent.ts.
• Each provider maps to a specific system prompt function (getSystemPrompt, getSystemPromptV1_5, getSystemPromptDoubao_15_15B, or getSystemPromptDoubao_15_20B) that defines the action grammar.
• The parser in multimodal/omni-tars/core/src/utils/parser.ts validates LLM output against the grammar specified by the active system prompt.
• Vision-language providers (e.g., doubao_1_5_vl) enable multimodal action parsing through provider-specific prompt extensions.

Frequently Asked Questions

What happens if I switch VLM providers without restarting the application?

UI-TARS-desktop supports runtime provider switching. When you change the provider, the application calls getModelVersion() and getSpByModelVersion() to regenerate the system prompt dynamically, allowing the parser to immediately recognize the new provider's action grammar without requiring a restart.

Why does the Doubao 1.5 VL provider require different parsing logic?

The VLMProviderV2.doubao_1_5_vl provider resolves to UITarsModelVersion.DOUBAO_1_5_20B, which loads getSystemPromptDoubao_15_20B(). This system prompt includes additional instructions for handling image-based tool calls and vision-language features that the standard parseComputerContent function in multimodal/omni-tars/core/src/utils/parser.ts is configured to recognize, whereas text-only providers would reject or ignore these multimodal action formats.

Can I add custom VLM providers to UI-TARS-desktop?

The current implementation in apps/ui-tars/src/main/utils/agent.ts uses a fixed switch statement for provider-to-model mapping. To add a custom provider, you would need to extend the VLMProviderV2 enum in apps/ui-tars/src/main/store/types.ts, add a corresponding case to getModelVersion() (lines 23-37), and implement a matching system prompt generator function called within getSpByModelVersion() (lines 40-54).

How do environment variables configure VLM providers?

The provider can be configured via environment variables such as VLM_PROVIDER and VLM_BASE_URL, as demonstrated in packages/ui-tars/operators/browser-operator/examples/default.ts. These variables determine which provider enum value is instantiated at startup, which then triggers the provider-specific model versioning and system prompt selection logic described in apps/ui-tars/src/main/utils/agent.ts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}