# UI-TARS Coordinate Transformation: 3 Factors Driving Normalization Calculations

> Discover the 3 key factors driving UI-TARS coordinate transformation calculations: raw input, NormalizeCoordinates function, and screenshot dimensions for accurate scaling.

- Repository: [Bytedance Inc./UI-TARS-desktop](https://github.com/bytedance/UI-TARS-desktop)
- Tags: deep-dive
- Published: 2026-05-10

---

**UI-TARS performs coordinate transformation using three key factors: raw input coordinates from action objects, a customizable `NormalizeCoordinates` function, and screenshot dimensions that serve as the reference frame for scaling.**

The `bytedance/UI-TARS-desktop` repository implements a robust normalization pipeline for GUI agent actions. Understanding the **coordinate transformation calculations** in UI-TARS is essential for anyone building or debugging UI automation, as it determines how absolute pixel positions are converted into normalized values relative to the captured screen size.

## The Three Factors Driving Coordinate Transformation

UI-TARS calculates transformed coordinates by combining three distinct inputs that work together to produce device-independent action data.

### 1. Raw Input Coordinates from BaseAction

The process begins with the raw positional data stored in a `BaseAction.inputs` object. According to the source code in [`multimodal/gui-agent/shared/src/utils/coordinateNormalizer.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/multimodal/gui-agent/shared/src/utils/coordinateNormalizer.ts) (lines 21‑46), the system extracts three potential coordinate fields:

- **`point`** – single-click or tap locations
- **`start`** – beginning coordinates for drag operations
- **`end`** – destination coordinates for drag operations

The `normalizeActionCoords` helper extracts whichever fields are present and prepares them for transformation.

### 2. The Normalization Function Implementation

The second factor is a pluggable normalization strategy defined by the `NormalizeCoordinates` type in [`multimodal/gui-agent/shared/src/types.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/multimodal/gui-agent/shared/src/types.ts). This function receives a `Coordinates` object and returns a structure containing the transformed result:

```typescript
// Simplified signature from the source
type NormalizeCoordinates = (coord: Coordinates) => {
  normalized: Coordinates;
  // optional metadata like scaling factors
};

```

In [`coordinateNormalizer.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/coordinateNormalizer.ts) (lines 27‑28, 37‑38, 44‑45), the system invokes this function for each extracted coordinate set, replacing the original values with the normalized output.

### 3. Screenshot Context and Screen Dimensions

The third critical factor is the screenshot size metadata that provides the reference frame for scaling. In [`packages/ui-tars/visualizer/src/transform.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/packages/ui-tars/visualizer/src/transform.ts) (line 49), the dump builder attaches screen dimensions via `conv.screenshotContext?.size`:

```typescript
pageContext: {
  size: conv.screenshotContext?.size || { width: 0, height: 0 },
  screenshotBase64: addBase64ImagePrefix(conv.screenshotBase64 ?? ''),
}

```

These dimensions (width × height) enable the normalizer to calculate scale factors (pixel ÷ screen‑width or pixel ÷ screen‑height) and optionally account for device‑pixel‑ratio (DPR) to produce device‑independent coordinates.

## How Coordinates Are Normalized in Practice

The transformation flow combines these factors inside the `normalizeActionCoords` utility. Here is a practical implementation that converts absolute pixel positions to percentage-based coordinates relative to the screenshot dimensions:

```typescript
import { normalizeActionCoords } from '@/shared/src/utils/coordinateNormalizer';
import { type BaseAction, type Coordinates, type NormalizeCoordinates } from '@/shared/src/types';

// Normalizer that converts pixels to percentages of screen size
const percentNormaliser: NormalizeCoordinates = (coord: Coordinates) => ({
  normalized: {
    x: coord.x / screenWidth,
    y: coord.y / screenHeight,
  },
});

// Raw action from the GUI agent
const rawAction: BaseAction = {
  type: 'click',
  inputs: { point: { x: 320, y: 240 } },
  // ... other action properties
};

// Transformed action with normalized coordinates
const normalisedAction = normalizeActionCoords(rawAction, percentNormaliser);
// Result: normalisedAction.inputs.point → { x: 0.4, y: 0.48 } 
// (assuming an 800×500 screenshot reference)

```

## Key Source Files for Coordinate Transformation

| File | Role |
|------|------|
| [`packages/ui-tars/visualizer/src/transform.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/packages/ui-tars/visualizer/src/transform.ts) | Builds the execution dump and attaches screenshot size (`size` field) used as the coordinate reference frame. |
| [`multimodal/gui-agent/shared/src/utils/coordinateNormalizer.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/multimodal/gui-agent/shared/src/utils/coordinateNormalizer.ts) | Contains the `normalizeActionCoords` helper that applies normalization to `point`, `start`, and `end` coordinates. |
| [`multimodal/gui-agent/shared/src/types.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/multimodal/gui-agent/shared/src/types.ts) | Defines `BaseAction`, `Coordinates`, and the `NormalizeCoordinates` type signature. |

## Summary

- **Raw coordinates** (`point`, `start`, `end`) are extracted from `BaseAction.inputs` as the initial values for transformation.
- A **normalization function** (`NormalizeCoordinates`) is applied to convert these values, typically scaling them relative to screen dimensions.
- **Screenshot context** provides the reference dimensions (width/height) needed to calculate accurate scale factors and ensure device-independent coordinates.
- The `normalizeActionCoords` utility in [`coordinateNormalizer.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/coordinateNormalizer.ts) orchestrates the entire transformation pipeline.

## Frequently Asked Questions

### What is the purpose of coordinate normalization in UI-TARS?

Normalization ensures that UI action coordinates are stored as relative values rather than absolute pixels. This makes recordings portable across devices with different screen resolutions and allows the agent to replay actions accurately regardless of display scaling.

### How does the NormalizeCoordinates function receive screen dimensions?

The function itself receives only the `Coordinates` object. However, the implementing closure can capture the screenshot dimensions from the `pageContext.size` field (populated in [`transform.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/transform.ts)), allowing it to calculate percentages or scaled values relative to the specific screen capture that generated the action.

### Where is the screenshot size captured in the UI-TARS pipeline?

The size is captured in [`packages/ui-tars/visualizer/src/transform.ts`](https://github.com/bytedance/UI-TARS-desktop/blob/main/packages/ui-tars/visualizer/src/transform.ts) during the dump generation process. Specifically, line 49 extracts `conv.screenshotContext?.size` and embeds it in the task's `pageContext`, making it available to downstream normalization logic.

### Can custom normalization logic be applied to UI-TARS actions?

Yes. The architecture accepts any function matching the `NormalizeCoordinates` type signature. Developers can inject custom logic—such as accounting for browser zoom levels, specific DPI settings, or coordinate offsets—by providing their own implementation to the `normalizeActionCoords` helper.