# How Cua Handles Multi-Touch Gestures and Mobile-Specific Interactions

> Discover how Cua handles multi-touch gestures and mobile interactions with its transport-agnostic architecture and high-level async API. Supports ADB and gRPC.

- Repository: [Cua/cua](https://github.com/trycua/cua)
- Tags: internals
- Published: 2026-04-27

---

**Cua implements multi-touch gestures and mobile interactions through a transport-agnostic architecture that abstracts Android input commands into a high-level async API, supporting both local ADB and remote gRPC emulator connections.**

The **Cua** framework (available at `trycua/cua`) provides a Python-based automation layer for Android devices that cleanly separates gesture intent from low-level event injection. Its mobile interface allows developers to execute complex multi-touch gestures—from simple taps to N-finger swipes—while the transport layer handles the device-specific implementation via Android Debug Bridge (ADB) or remote emulator protocols.

## Transport-Agnostic Architecture for Mobile Automation

Cua’s mobile stack is organized into three distinct layers that isolate device communication from gesture logic:

- **High-Level API** – The `Mobile` class in [`libs/python/cua-sandbox/cua_sandbox/interfaces/mobile.py`](https://github.com/trycua/cua/blob/main/libs/python/cua-sandbox/cua_sandbox/interfaces/mobile.py) exposes async methods for taps, swipes, pinches, and hardware keys.
- **Transport Abstraction** – The `Transport` base class in [`libs/python/cua-sandbox/cua_sandbox/transport/base.py`](https://github.com/trycua/cua/blob/main/libs/python/cua-sandbox/cua_sandbox/transport/base.py) defines the contract for sending commands, with concrete implementations for ADB and gRPC.
- **Multi-Touch Injection** – Low-level `sendevent` sequences implementing Android’s **MT Protocol B** are generated in [`libs/python/cua-sandbox/cua_sandbox/transport/adb.py`](https://github.com/trycua/cua/blob/main/libs/python/cua-sandbox/cua_sandbox/transport/adb.py) for local devices, and mirrored in [`libs/python/cua-sandbox/cua_sandbox/transport/grpc_emulator.py`](https://github.com/trycua/cua/blob/main/libs/python/cua-sandbox/cua_sandbox/transport/grpc_emulator.py) for remote cloud emulators.

## The Mobile Class and High-Level Gesture API

The **`Mobile`** class serves as the primary interface for mobile-specific interactions, wrapping transport-specific details into intuitive async methods.

### Single-Touch Operations

Standard single-pointer actions execute via `adb shell input` commands:

- **`tap(x, y)`** – Quick touch and release at screen coordinates.
- **`long_press(x, y, duration_ms)`** – Extended press with configurable hold time.
- **`swipe(x1, y1, x2, y2, duration_ms)`** – Linear drag operation.
- **`scroll_*` and `fling`** – Momentum-based scrolling utilities.

Before executing gestures, the class queries screen dimensions via `wm size` to translate logical pixel coordinates to the device’s native resolution.

### Generic N-Finger Gestures

For complex **multi-touch gestures**, the `gesture()` method accepts an arbitrary number of finger paths and forwards a structured payload to the transport layer:

```python
await mobile.gesture(
    (cx - 20, cy), (cx - 200, cy),   # finger 0: start → end

    (cx + 20, cy), (cx + 200, cy),   # finger 1: start → end

    duration_ms=400
)

```

The method validates path arguments, constructs per-finger coordinate pairs, and sends the `"multitouch_gesture"` action to the transport with `screen_w`, `screen_h`, and interpolation `steps` parameters.

### Convenience Helpers for Pinch Gestures

**`pinch_in`** and **`pinch_out`** are thin wrappers around `gesture()` that calculate two-finger start and end points centered on `(cx, cy)`:

```python
await mobile.pinch_out(cx=540, cy=960, spread=250, duration_ms=500)

```

## Low-Level Multi-Touch Implementation via ADB

When `Mobile` dispatches a `"multitouch_gesture"` action, **`ADBTransport._multitouch_gesture`** in [`libs/python/cua-sandbox/cua_sandbox/transport/adb.py`](https://github.com/trycua/cua/blob/main/libs/python/cua-sandbox/cua_sandbox/transport/adb.py) handles the kernel-level event injection:

1. **Root Access** – Executes `adb root` to ensure sufficient privileges for `/dev/input/event*` access.
2. **Input Node Discovery** – Identifies the correct touch device by scanning for `ABS_MT_POSITION_X` (code `0x0035`) capability reports.
3. **Coordinate Mapping** – Reads the device’s axis maximum (typically `32767`) to create a `px_to_raw` scaling function.
4. **MT Protocol B Sequence Generation** – Constructs a chain of `sendevent` commands:
   - Assigns **slots** (`ABS_MT_SLOT`) and **tracking IDs** (`ABS_MT_TRACKING_ID`) for each finger.
   - Sets **X/Y coordinates** (`ABS_MT_POSITION_X/Y`) and **pressure** values.
   - Emits **synchronization events** (`SYN_REPORT`) and `BTN_TOUCH = 1` for the initial contact.
   - Interpolates intermediate positions for smooth movement across the specified `duration_ms`.
   - Releases contacts by setting `TID_NONE` and `BTN_TOUCH = 0`, followed by a final sync.
5. **Batch Execution** – Joins all commands with `&&` and executes via a single `adb shell` invocation.

This approach implements the **MT Protocol B** standard that the Android kernel interprets as simultaneous touch contacts, enabling true multi-touch simulation rather than sequential single-touch events.

## Remote Emulator and gRPC Transport

For cloud-based testing environments, **`GrpcEmulatorTransport`** in [`libs/python/cua-sandbox/cua_sandbox/transport/grpc_emulator.py`](https://github.com/trycua/cua/blob/main/libs/python/cua-sandbox/cua_sandbox/transport/grpc_emulator.py) accepts the same gesture payload structure and forwards it via gRPC to a remote emulator server.

The server-side handler in [`libs/python/computer-server/computer_server/handlers/android.py`](https://github.com/trycua/cua/blob/main/libs/python/computer-server/computer_server/handlers/android.py) receives the `multitouch_gesture` RPC and executes identical `sendevent` logic within the emulator’s ADB environment. This ensures that high-level API calls work transparently across both local USB-connected devices and remote cloud emulators without code changes.

## Mobile-Specific Interactions Beyond Touch

Cua handles non-gesture mobile interactions through the same transport abstraction:

- **Hardware Keys** – `home()`, `back()`, `recents()`, `power()`, and volume controls map to `input keyevent <code>` shell commands.
- **Text Input** – `enter()` and `backspace()` methods send specific keyevent codes.
- **System Actions** – `wake()`, `notifications()`, and `close_notifications()` use either key events or direct `service call` invocations to manipulate the Android system UI.

All methods are **asynchronous** and share the transport layer, ensuring consistent behavior across ADB and remote back-ends.

## Practical Implementation Examples

### Basic Single-Touch Tap

```python
from cua_sandbox.transport.adb import ADBTransport
from cua_sandbox.interfaces.mobile import Mobile

transport = ADBTransport(serial="emulator-5554")
await transport.connect()
mobile = Mobile(transport)

await mobile.tap(300, 600)          # tap at (300, 600) screen pixels

await transport.disconnect()

```

### Two-Finger Pinch-Out (Zoom-In)

```python
await mobile.pinch_out(cx=540, cy=960, spread=250, duration_ms=500)

```

The `cx` and `cy` parameters specify the pinch center, while `spread` determines the distance each finger travels from that center.

### Arbitrary Three-Finger Swipe

```python
await mobile.gesture(
    (100, 100), (100, 800),   # Finger 0

    (300, 100), (300, 800),   # Finger 1

    (500, 100), (500, 800),   # Finger 2

    duration_ms=800,
)

```

### Remote Cloud Emulator Usage

```python
from cua_sandbox.transport.grpc_emulator import GrpcEmulatorTransport

transport = GrpcEmulatorTransport(host="emu.example.com", token="…")
await transport.connect()
mobile = Mobile(transport)

await mobile.pinch_in(cx=540, cy=960, spread=200)
await transport.disconnect()

```

## Summary

- **Cua** provides **transport-agnostic mobile automation** through the `Mobile` class, supporting both ADB and gRPC transports.
- **Multi-touch gestures** are handled via the `gesture()` method, which supports arbitrary N-finger paths with configurable duration and interpolation.
- **`ADBTransport._multitouch_gesture`** implements **MT Protocol B** `sendevent` sequences to inject true simultaneous touch events at the kernel level.
- **Remote emulators** receive identical gesture payloads via gRPC, with server-side execution matching local ADB behavior.
- All interactions—including hardware keys and system actions—are **async** and work uniformly across local and cloud Android targets.

## Frequently Asked Questions

### How does Cua distinguish between single-touch and multi-touch gestures?

Single-touch actions like `tap()` and `swipe()` use standard `adb shell input` commands, while multi-touch gestures invoke the `gesture()` method which constructs a complex payload containing multiple finger paths. The transport layer detects this payload type and routes it to `_multitouch_gesture` for low-level `sendevent` injection rather than simple shell input commands.

### What Android permissions are required for Cua’s multi-touch injection?

The device must allow `adb root` access because the implementation writes directly to `/dev/input/event*` nodes using the `sendevent` command. This requires root privileges to bypass standard Android input stack restrictions and inject raw MT Protocol B events at the kernel driver level.

### Can Cua handle gestures on emulators as well as physical devices?

Yes. The architecture supports both physical devices via `ADBTransport` and remote emulators via `GrpcEmulatorTransport`. The `GrpcEmulatorTransport` forwards the same gesture payload to a cloud server running [`computer_server/handlers/android.py`](https://github.com/trycua/cua/blob/main/computer_server/handlers/android.py), which executes identical `sendevent` logic inside the emulator’s environment, ensuring consistent behavior across deployment targets.

### How does Cua calculate intermediate positions for smooth gestures?

When `steps` is set to `0` (auto-compute) or a specific value, the transport implementation interpolates between start and end coordinates for each finger across the `duration_ms` timeframe. It generates discrete `sendevent` commands for each intermediate position, creating smooth linear movement that mimics natural touch input rather than instantaneous teleportation between points.