UI-TARS Coordinate Transformation: 3 Factors Driving Normalization Calculations

UI-TARS performs coordinate transformation using three key factors: raw input coordinates from action objects, a customizable NormalizeCoordinates function, and screenshot dimensions that serve as the reference frame for scaling.

The bytedance/UI-TARS-desktop repository implements a robust normalization pipeline for GUI agent actions. Understanding the coordinate transformation calculations in UI-TARS is essential for anyone building or debugging UI automation, as it determines how absolute pixel positions are converted into normalized values relative to the captured screen size.

The Three Factors Driving Coordinate Transformation

UI-TARS calculates transformed coordinates by combining three distinct inputs that work together to produce device-independent action data.

1. Raw Input Coordinates from BaseAction

The process begins with the raw positional data stored in a BaseAction.inputs object. According to the source code in multimodal/gui-agent/shared/src/utils/coordinateNormalizer.ts (lines 21‑46), the system extracts three potential coordinate fields:

• point – single-click or tap locations
• start – beginning coordinates for drag operations
• end – destination coordinates for drag operations

The normalizeActionCoords helper extracts whichever fields are present and prepares them for transformation.

2. The Normalization Function Implementation

The second factor is a pluggable normalization strategy defined by the NormalizeCoordinates type in multimodal/gui-agent/shared/src/types.ts. This function receives a Coordinates object and returns a structure containing the transformed result:

// Simplified signature from the source
type NormalizeCoordinates = (coord: Coordinates) => {
  normalized: Coordinates;
  // optional metadata like scaling factors
};

In coordinateNormalizer.ts (lines 27‑28, 37‑38, 44‑45), the system invokes this function for each extracted coordinate set, replacing the original values with the normalized output.

3. Screenshot Context and Screen Dimensions

The third critical factor is the screenshot size metadata that provides the reference frame for scaling. In packages/ui-tars/visualizer/src/transform.ts (line 49), the dump builder attaches screen dimensions via conv.screenshotContext?.size:

pageContext: {
  size: conv.screenshotContext?.size || { width: 0, height: 0 },
  screenshotBase64: addBase64ImagePrefix(conv.screenshotBase64 ?? ''),
}

These dimensions (width × height) enable the normalizer to calculate scale factors (pixel ÷ screen‑width or pixel ÷ screen‑height) and optionally account for device‑pixel‑ratio (DPR) to produce device‑independent coordinates.

How Coordinates Are Normalized in Practice

The transformation flow combines these factors inside the normalizeActionCoords utility. Here is a practical implementation that converts absolute pixel positions to percentage-based coordinates relative to the screenshot dimensions:

import { normalizeActionCoords } from '@/shared/src/utils/coordinateNormalizer';
import { type BaseAction, type Coordinates, type NormalizeCoordinates } from '@/shared/src/types';

// Normalizer that converts pixels to percentages of screen size
const percentNormaliser: NormalizeCoordinates = (coord: Coordinates) => ({
  normalized: {
    x: coord.x / screenWidth,
    y: coord.y / screenHeight,
  },
});

// Raw action from the GUI agent
const rawAction: BaseAction = {
  type: 'click',
  inputs: { point: { x: 320, y: 240 } },
  // ... other action properties
};

// Transformed action with normalized coordinates
const normalisedAction = normalizeActionCoords(rawAction, percentNormaliser);
// Result: normalisedAction.inputs.point → { x: 0.4, y: 0.48 } 
// (assuming an 800×500 screenshot reference)

Key Source Files for Coordinate Transformation

File	Role
packages/ui-tars/visualizer/src/transform.ts	Builds the execution dump and attaches screenshot size (size field) used as the coordinate reference frame.
multimodal/gui-agent/shared/src/utils/coordinateNormalizer.ts	Contains the normalizeActionCoords helper that applies normalization to point, start, and end coordinates.
multimodal/gui-agent/shared/src/types.ts	Defines BaseAction, Coordinates, and the NormalizeCoordinates type signature.

Summary

• Raw coordinates (point, start, end) are extracted from BaseAction.inputs as the initial values for transformation.
• A normalization function (NormalizeCoordinates) is applied to convert these values, typically scaling them relative to screen dimensions.
• Screenshot context provides the reference dimensions (width/height) needed to calculate accurate scale factors and ensure device-independent coordinates.
• The normalizeActionCoords utility in coordinateNormalizer.ts orchestrates the entire transformation pipeline.

Frequently Asked Questions

What is the purpose of coordinate normalization in UI-TARS?

Normalization ensures that UI action coordinates are stored as relative values rather than absolute pixels. This makes recordings portable across devices with different screen resolutions and allows the agent to replay actions accurately regardless of display scaling.

How does the NormalizeCoordinates function receive screen dimensions?

The function itself receives only the Coordinates object. However, the implementing closure can capture the screenshot dimensions from the pageContext.size field (populated in transform.ts), allowing it to calculate percentages or scaled values relative to the specific screen capture that generated the action.

Where is the screenshot size captured in the UI-TARS pipeline?

The size is captured in packages/ui-tars/visualizer/src/transform.ts during the dump generation process. Specifically, line 49 extracts conv.screenshotContext?.size and embeds it in the task's pageContext, making it available to downstream normalization logic.

Can custom normalization logic be applied to UI-TARS actions?

Yes. The architecture accepts any function matching the NormalizeCoordinates type signature. Developers can inject custom logic—such as accounting for browser zoom levels, specific DPI settings, or coordinate offsets—by providing their own implementation to the normalizeActionCoords helper.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}