How Scale Factors Impact Coordinate Mapping in UI-TARS Desktop

Scale factors in UI-TARS Desktop convert logical model coordinates to physical screen coordinates by multiplying predicted values by the device pixel ratio (DPR), ensuring automation actions land precisely on high-DPI displays.

UI-TARS Desktop bridges high-level language model predictions with low-level OS automation, and correctly handling scale factors is critical for accurate coordinate mapping between the logical space where the AI reasons and the physical pixels where actions execute. The repository implements a unified scaleFactor field that travels from screenshot capture through action parsing to final execution.

The Role of Device Pixel Ratio in Screenshot Metadata

Modern displays use a device pixel ratio (DPR)—the number of physical pixels per logical CSS pixel—to render crisp interfaces on high-DPI screens. UI-TARS Desktop records this ratio as scaleFactor in image metadata, creating a conversion layer between the model's predictions and the host operating system.

Where Scale Factors Are Defined

The screenshot payload type defines an optional scaleFactor property in packages/ui-tars/shared/src/types/data.ts at line 25:

scaleFactor?: number;

This definition establishes the contract for DPR values throughout the system. The SDK type in packages/ui-tars/sdk/src/types.ts at line 27 propagates this value, ensuring every component that receives a screenshot can access the scaling ratio directly.

Converting Logical Coordinates to Physical Actions

When the LLM returns actions like click(start_box=[100,200,300,400]), these coordinates exist in logical space—the normalized coordinates the model predicts based on the screenshot dimensions. Before execution, the system must convert these to physical pixels that match the actual display hardware.

Parsing Model Outputs with Scale Factors

The action parser in packages/ui-tars/action-parser/src/actionParser.ts (lines 191-200) handles this conversion by multiplying each coordinate by the provided scaleFactor, defaulting to 1 if absent:

const floatNumbers = numbers.map((num, idx) =>
  parseFloat(num) * (scaleFactor ?? 1)
);

This multiplication transforms logical boxes into physical coordinates that underlying automation libraries can execute directly on screen.

Practical Coordinate Scaling Example

Here is how the conversion works in practice:

function scaleBox(box: [number, number, number, number], scaleFactor = 1) {
  // box = [x0, y0, x1, y1] in logical pixels
  return box.map(v => v * scaleFactor) as [number, number, number, number];
}

// Example usage:
const logicalBox = [120, 80, 240, 160];
const scaleFactor = 2;               // e.g., Retina display (2×)
const physicalBox = scaleBox(logicalBox, scaleFactor);
// physicalBox => [240, 160, 480, 320]

Operator Implementation Across Platforms

Different operators compute the scaleFactor from their respective capture mechanisms, ensuring consistency across native and browser environments.

Native Screen Capture (nut-js Operator)

The nut-js operator retrieves DPR from image pixel density metadata in packages/ui-tars/operators/nut-js/src/index.ts (lines 55-64):

const screenWithScale = await grabImage.toRGB();
const scaleFactor = screenWithScale.pixelDensity.scaleX; // DPR from capture
logger.info('[NutjsOperator] screenshot: '
  + `${screenWithScale.width / scaleFactor}x${screenWithScale.height / scaleFactor}`
  + `, scaleFactor: ${scaleFactor}`);

Here, pixelDensity.scaleX (and scaleY) provides the exact ratio needed to map the model's logical coordinates to the physical display.

Browser Automation (browser-operator)

For Electron-based browser automation, the operator queries the viewport's device scale factor in packages/ui-tars/operators/browser-operator/src/browser-operator.ts (lines 681-684):

const scaleFactor = page.viewport()?.deviceScaleFactor ?? 1;
this.deviceScaleFactor = scaleFactor;
logger.debug('getDeviceScaleFactor: deviceScaleFactor: ', scaleFactor);

Both implementations store this value as scaleFactor (or deviceScaleFactor) in the screenshot metadata, maintaining the contract defined in the shared types.

Why Coordinate Scaling Matters for Automation

Without proper scale factor handling, all coordinate-based actions—clicks, drags, swipes, and scrolls—would deviate by the DPR multiplier on high-DPI displays like Retina MacBooks or 4K monitors. For example, on a display with DPR of 2, a click intended for position (100,100) would land at (50,50) in logical space if unscaled, completely missing the target element.

The end-to-end flow ensures precision:

1. Screenshot capture records the scaleFactor from the OS or browser viewport.
2. The model predicts actions using logical coordinates relative to the screenshot.
3. actionParser multiplies coordinates by scaleFactor before building ActionInputs.
4. The operator receives physical coordinates and executes the command at the exact screen location.

Summary

• Scale factors in UI-TARS Desktop represent the device pixel ratio (DPR) that converts between logical and physical coordinate spaces.
• The scaleFactor field is defined in packages/ui-tars/shared/src/types/data.ts and propagated through the SDK types.
• The action parser at packages/ui-tars/action-parser/src/actionParser.ts multiplies model coordinates by the scale factor to produce physical pixel values.
• Native operators extract DPR from pixelDensity.scaleX, while browser operators use Electron's deviceScaleFactor.
• Ignoring scale factors causes misaligned automation actions on high-DPI displays, with offsets proportional to the DPR value.

Frequently Asked Questions

What happens if scaleFactor is missing or set to 1 on a Retina display?

If scaleFactor defaults to 1 on a Retina display (where DPR is typically 2), the automation will click at half the intended physical coordinates. For example, a target at logical coordinates (200, 200) would receive input at physical coordinates (200, 200) instead of (400, 400), causing the action to miss the element entirely.

Why does UI-TARS use logical coordinates instead of physical pixels directly?

Language models reason about UI elements in normalized, logical space relative to the screenshot dimensions, which remains consistent across different display densities. Using logical coordinates allows the same model to work on both standard and high-DPI displays without retraining, with the client-side scaleFactor handling the final conversion to physical pixels.

How do I verify the correct scale factor is being detected?

Check the operator logs for the scaleFactor or deviceScaleFactor values. In the nut-js operator, look for the log line containing [NutjsOperator] screenshot: which outputs the calculated dimensions and scale factor. For browser operations, check the debug log for getDeviceScaleFactor: deviceScaleFactor:. These values should match your display's DPR (1 for standard, 2 for Retina, etc.).

Does this affect all action types or only click coordinates?

All coordinate-based actions are affected, including clicks, drag operations, swipe gestures, and element boundary boxes. Any action that receives start_box or coordinate parameters from the model passes through the same scaling logic in actionParser.ts before execution.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}