# How Scale Factors Impact Coordinate Mapping in UI-TARS Desktop

> Discover how scale factors in UI-TARS Desktop map logical coordinates to physical screen coordinates using DPR for precise automation on high-DPI displays.

- Repository: [Bytedance Inc./UI-TARS-desktop](https://github.com/bytedance/UI-TARS-desktop)
- Tags: deep-dive
- Published: 2026-05-10

---

**Scale factors in UI-TARS Desktop convert logical model coordinates to physical screen coordinates by multiplying predicted values by the device pixel ratio (DPR), ensuring automation actions land precisely on high-DPI displays.**

UI-TARS Desktop bridges high-level language model predictions with low-level OS automation, and correctly handling **scale factors** is critical for accurate coordinate mapping between the logical space where the AI reasons and the physical pixels where actions execute. The repository implements a unified `scaleFactor` field that travels from screenshot capture through action parsing to final execution.

## The Role of Device Pixel Ratio in Screenshot Metadata

Modern displays use a **device pixel ratio** (DPR)—the number of physical pixels per logical CSS pixel—to render crisp interfaces on high-DPI screens. UI-TARS Desktop records this ratio as `scaleFactor` in image metadata, creating a conversion layer between the model's predictions and the host operating system.

### Where Scale Factors Are Defined

The screenshot payload type defines an optional `scaleFactor` property in [`packages/ui-tars/shared/src/types/data.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/packages/ui-tars/shared/src/types/data.ts) at line 25:

```typescript
scaleFactor?: number;

```

This definition establishes the contract for DPR values throughout the system. The SDK type in [`packages/ui-tars/sdk/src/types.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/packages/ui-tars/sdk/src/types.ts) at line 27 propagates this value, ensuring every component that receives a screenshot can access the scaling ratio directly.

## Converting Logical Coordinates to Physical Actions

When the LLM returns actions like `click(start_box=[100,200,300,400])`, these coordinates exist in **logical space**—the normalized coordinates the model predicts based on the screenshot dimensions. Before execution, the system must convert these to **physical pixels** that match the actual display hardware.

### Parsing Model Outputs with Scale Factors

The action parser in [`packages/ui-tars/action-parser/src/actionParser.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/packages/ui-tars/action-parser/src/actionParser.ts) (lines 191-200) handles this conversion by multiplying each coordinate by the provided `scaleFactor`, defaulting to `1` if absent:

```typescript
const floatNumbers = numbers.map((num, idx) =>
  parseFloat(num) * (scaleFactor ?? 1)
);

```

This multiplication transforms logical boxes into physical coordinates that underlying automation libraries can execute directly on screen.

### Practical Coordinate Scaling Example

Here is how the conversion works in practice:

```typescript
function scaleBox(box: [number, number, number, number], scaleFactor = 1) {
  // box = [x0, y0, x1, y1] in logical pixels
  return box.map(v => v * scaleFactor) as [number, number, number, number];
}

// Example usage:
const logicalBox = [120, 80, 240, 160];
const scaleFactor = 2;               // e.g., Retina display (2×)
const physicalBox = scaleBox(logicalBox, scaleFactor);
// physicalBox => [240, 160, 480, 320]

```

## Operator Implementation Across Platforms

Different operators compute the `scaleFactor` from their respective capture mechanisms, ensuring consistency across native and browser environments.

### Native Screen Capture (nut-js Operator)

The nut-js operator retrieves DPR from image pixel density metadata in [`packages/ui-tars/operators/nut-js/src/index.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/packages/ui-tars/operators/nut-js/src/index.ts) (lines 55-64):

```typescript
const screenWithScale = await grabImage.toRGB();
const scaleFactor = screenWithScale.pixelDensity.scaleX; // DPR from capture
logger.info('[NutjsOperator] screenshot: '
  + `${screenWithScale.width / scaleFactor}x${screenWithScale.height / scaleFactor}`
  + `, scaleFactor: ${scaleFactor}`);

```

Here, `pixelDensity.scaleX` (and `scaleY`) provides the exact ratio needed to map the model's logical coordinates to the physical display.

### Browser Automation (browser-operator)

For Electron-based browser automation, the operator queries the viewport's device scale factor in [`packages/ui-tars/operators/browser-operator/src/browser-operator.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/packages/ui-tars/operators/browser-operator/src/browser-operator.ts) (lines 681-684):

```typescript
const scaleFactor = page.viewport()?.deviceScaleFactor ?? 1;
this.deviceScaleFactor = scaleFactor;
logger.debug('getDeviceScaleFactor: deviceScaleFactor: ', scaleFactor);

```

Both implementations store this value as `scaleFactor` (or `deviceScaleFactor`) in the screenshot metadata, maintaining the contract defined in the shared types.

## Why Coordinate Scaling Matters for Automation

Without proper **scale factor** handling, all coordinate-based actions—clicks, drags, swipes, and scrolls—would deviate by the DPR multiplier on high-DPI displays like Retina MacBooks or 4K monitors. For example, on a display with DPR of 2, a click intended for position (100,100) would land at (50,50) in logical space if unscaled, completely missing the target element.

The end-to-end flow ensures precision:

1. Screenshot capture records the `scaleFactor` from the OS or browser viewport.
2. The model predicts actions using logical coordinates relative to the screenshot.
3. `actionParser` multiplies coordinates by `scaleFactor` before building `ActionInputs`.
4. The operator receives physical coordinates and executes the command at the exact screen location.

## Summary

- **Scale factors** in UI-TARS Desktop represent the device pixel ratio (DPR) that converts between logical and physical coordinate spaces.
- The `scaleFactor` field is defined in [`packages/ui-tars/shared/src/types/data.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/packages/ui-tars/shared/src/types/data.ts) and propagated through the SDK types.
- The action parser at [`packages/ui-tars/action-parser/src/actionParser.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/packages/ui-tars/action-parser/src/actionParser.ts) multiplies model coordinates by the scale factor to produce physical pixel values.
- Native operators extract DPR from `pixelDensity.scaleX`, while browser operators use Electron's `deviceScaleFactor`.
- Ignoring scale factors causes misaligned automation actions on high-DPI displays, with offsets proportional to the DPR value.

## Frequently Asked Questions

### What happens if scaleFactor is missing or set to 1 on a Retina display?

If `scaleFactor` defaults to 1 on a Retina display (where DPR is typically 2), the automation will click at half the intended physical coordinates. For example, a target at logical coordinates (200, 200) would receive input at physical coordinates (200, 200) instead of (400, 400), causing the action to miss the element entirely.

### Why does UI-TARS use logical coordinates instead of physical pixels directly?

Language models reason about UI elements in normalized, logical space relative to the screenshot dimensions, which remains consistent across different display densities. Using logical coordinates allows the same model to work on both standard and high-DPI displays without retraining, with the client-side `scaleFactor` handling the final conversion to physical pixels.

### How do I verify the correct scale factor is being detected?

Check the operator logs for the `scaleFactor` or `deviceScaleFactor` values. In the nut-js operator, look for the log line containing `[NutjsOperator] screenshot:` which outputs the calculated dimensions and scale factor. For browser operations, check the debug log for `getDeviceScaleFactor: deviceScaleFactor:`. These values should match your display's DPR (1 for standard, 2 for Retina, etc.).

### Does this affect all action types or only click coordinates?

All coordinate-based actions are affected, including clicks, drag operations, swipe gestures, and element boundary boxes. Any action that receives `start_box` or coordinate parameters from the model passes through the same scaling logic in [`actionParser.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/actionParser.ts) before execution.