# VLM Providers Available in UI-TARS Desktop and Their Impact on Action Parsing

> Discover the VLM providers in UI-TARS Desktop: Hugging Face and VolcEngine Ark. Learn how they impact action parsing and agent output grammar for better results.

- Repository: [Bytedance Inc./UI-TARS-desktop](https://github.com/bytedance/UI-TARS-desktop)
- Tags: deep-dive
- Published: 2026-05-10

---

**UI-TARS-desktop supports four distinct VLM providers—Hugging Face for UI-TARS 1.0/1.5 and VolcEngine Ark for Doubao 1.5 variants—which determine the system prompt loaded at runtime, thereby controlling the action-parsing grammar that structures the agent's output.**

The bytedance/UI-TARS-desktop repository implements a provider-agnostic agent architecture where the selected Vision-Language Model (VLM) provider dictates how natural language instructions translate into executable UI automation. Your provider selection triggers a cascade of configuration changes that determine both the model weights invoked and the structured action schema expected by the downstream parser.

## Available VLM Providers in UI-TARS

The available providers are defined as enum values in [`apps/ui-tars/src/main/store/types.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/apps/ui-tars/src/main/store/types.ts). UI-TARS-desktop currently supports four distinct configurations:

- **`VLMProviderV2.ui_tars_1_0`** – Hugging Face for UI-TARS-1.0
- **`VLMProviderV2.ui_tars_1_5`** – Hugging Face for UI-TARS-1.5  
- **`VLMProviderV2.doubao_1_5`** – VolcEngine Ark for Doubao-1.5-UI-TARS (15B)
- **`VLMProviderV2.doubao_1_5_vl`** – VolcEngine Ark for Doubao-1.5-thinking-vision-pro (20B)

Each enum value maps to a specific model version constant and a dedicated system prompt function that encodes the action-parsing grammar for that provider.

## How VLM Providers Control Action Parsing

The VLM provider affects action parsing through a two-step resolution process implemented in [`apps/ui-tars/src/main/utils/agent.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/apps/ui-tars/src/main/utils/agent.ts):

1. **Model Version Resolution** – The `getModelVersion()` function (lines 23-37) maps the provider enum to a concrete `UITarsModelVersion` constant.
2. **System Prompt Selection** – The `getSpByModelVersion()` function (lines 40-54) selects the provider-specific system prompt that defines the action grammar the LLM must follow.

The parser in [`multimodal/omni-tars/core/src/utils/parser.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/multimodal/omni-tars/core/src/utils/parser.ts)—specifically `parseComputerContent` (lines 181-200) and related functions—expects the model's output to conform to the schema described in the active system prompt. When you select a vision-language provider such as `doubao_1_5_vl`, the system prompt includes additional instructions for image-based tool calls, enabling the parser to recognize multimodal actions that standard providers cannot process.

### Provider-to-Model Version Mapping

In [`apps/ui-tars/src/main/utils/agent.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/apps/ui-tars/src/main/utils/agent.ts), the `getModelVersion` switch statement translates provider selection into concrete model versions:

- `ui_tars_1_0` → `UITarsModelVersion.V1_0`
- `ui_tars_1_5` → `UITarsModelVersion.V1_5`  
- `doubao_1_5` → `UITarsModelVersion.DOUBAO_1_5_15B`
- `doubao_1_5_vl` → `UITarsModelVersion.DOUBAO_1_5_20B`

This mapping ensures the agent invokes the correct endpoint weights for the selected provider.

### System Prompt Selection and Action Grammar

The `getSpByModelVersion` function returns one of four system prompt generators based on the resolved model version:

- **V1_0**: `getSystemPrompt()` – Default text-only action grammar
- **V1_5**: `getSystemPromptV1_5()` – Enhanced UI-TARS 1.5 grammar
- **DOUBAO_1_5_15B**: `getSystemPromptDoubao_15_15B()` – VolcEngine standard actions
- **DOUBAO_1_5_20B**: `getSystemPromptDoubao_15_20B()` – VolcEngine vision-language actions with image support

The system prompt encodes permissible tool calls, mouse event formats, and vision-language capabilities. Consequently, switching providers changes the set of action formats that `parseComputerContent` can successfully validate and execute.

## Configuring VLM Providers: Code Examples

### Selecting a Provider and Loading the Correct Prompt

The following TypeScript example demonstrates how provider selection drives model version and system prompt resolution:

```typescript
import { VLMProviderV2 } from '@/store/types';
import { getModelVersion, getSpByModelVersion } from '@/utils/agent';

// Select the Doubao 1.5 vision-language provider
const provider: VLMProviderV2 = VLMProviderV2.doubao_1_5_vl;

// Resolve to concrete model version: UITarsModelVersion.DOUBAO_1_5_20B
const modelVersion = getModelVersion(provider);

// Load the corresponding system prompt with action grammar
const systemPrompt = getSpByModelVersion(
  modelVersion,
  'en',          // language code
  'computer'     // operator type
);
// systemPrompt now contains Doubao-1.5-VL specific instructions

```

*Source: `getModelVersion` and `getSpByModelVersion` are defined in [`apps/ui-tars/src/main/utils/agent.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/apps/ui-tars/src/main/utils/agent.ts) (lines 23-54).*

### Parsing Model Output According to Provider Grammar

The parser expects tool definitions that match the active system prompt's schema:

```typescript
import { parseComputerContent } from '@/multimodal/omni-tars/core/src/utils/parser';

const rawResponse = `{
  "content": "move mouse to (400,300) and left_click",
  "tools": [{
    "type": "function",
    "function": {
      "name": "mouse_move",
      "arguments": "{\"x\":400,\"y\":300}"
    }
  },{
    "type": "function",
    "function": {
      "name":"left_click",
      "arguments":"{}"
    }
  }]
}`;

const parsed = parseComputerContent(rawResponse);
// parsed.tools contains structured actions for execution

```

*Source: Action parsing logic resides in [`multimodal/omni-tars/core/src/utils/parser.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/multimodal/omni-tars/core/src/utils/parser.ts) (lines 181-200).*

### Runtime Provider Switching

You can reconfigure the agent at runtime by updating the provider and regenerating the system prompt:

```typescript
function onProviderChange(newProvider: VLMProviderV2) {
  // Re-compute model version and prompt for new provider
  const version = getModelVersion(newProvider);
  const prompt = getSpByModelVersion(version, 'en', 'computer');
  
  // Update agent configuration
  agent.setSystemPrompt(prompt);
}

```

## Summary

- UI-TARS-desktop supports four VLM providers via the `VLMProviderV2` enum defined in [`apps/ui-tars/src/main/store/types.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/apps/ui-tars/src/main/store/types.ts).
- Provider selection determines the model version through `getModelVersion()` in [`apps/ui-tars/src/main/utils/agent.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/apps/ui-tars/src/main/utils/agent.ts).
- Each provider maps to a specific system prompt function (`getSystemPrompt`, `getSystemPromptV1_5`, `getSystemPromptDoubao_15_15B`, or `getSystemPromptDoubao_15_20B`) that defines the action grammar.
- The parser in [`multimodal/omni-tars/core/src/utils/parser.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/multimodal/omni-tars/core/src/utils/parser.ts) validates LLM output against the grammar specified by the active system prompt.
- Vision-language providers (e.g., `doubao_1_5_vl`) enable multimodal action parsing through provider-specific prompt extensions.

## Frequently Asked Questions

### What happens if I switch VLM providers without restarting the application?

UI-TARS-desktop supports runtime provider switching. When you change the provider, the application calls `getModelVersion()` and `getSpByModelVersion()` to regenerate the system prompt dynamically, allowing the parser to immediately recognize the new provider's action grammar without requiring a restart.

### Why does the Doubao 1.5 VL provider require different parsing logic?

The `VLMProviderV2.doubao_1_5_vl` provider resolves to `UITarsModelVersion.DOUBAO_1_5_20B`, which loads `getSystemPromptDoubao_15_20B()`. This system prompt includes additional instructions for handling image-based tool calls and vision-language features that the standard `parseComputerContent` function in [`multimodal/omni-tars/core/src/utils/parser.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/multimodal/omni-tars/core/src/utils/parser.ts) is configured to recognize, whereas text-only providers would reject or ignore these multimodal action formats.

### Can I add custom VLM providers to UI-TARS-desktop?

The current implementation in [`apps/ui-tars/src/main/utils/agent.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/apps/ui-tars/src/main/utils/agent.ts) uses a fixed switch statement for provider-to-model mapping. To add a custom provider, you would need to extend the `VLMProviderV2` enum in [`apps/ui-tars/src/main/store/types.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/apps/ui-tars/src/main/store/types.ts), add a corresponding case to `getModelVersion()` (lines 23-37), and implement a matching system prompt generator function called within `getSpByModelVersion()` (lines 40-54).

### How do environment variables configure VLM providers?

The provider can be configured via environment variables such as `VLM_PROVIDER` and `VLM_BASE_URL`, as demonstrated in [`packages/ui-tars/operators/browser-operator/examples/default.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/packages/ui-tars/operators/browser-operator/examples/default.ts). These variables determine which provider enum value is instantiated at startup, which then triggers the provider-specific model versioning and system prompt selection logic described in [`apps/ui-tars/src/main/utils/agent.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/apps/ui-tars/src/main/utils/agent.ts).