How Cua Handles Multi-Touch Gestures and Mobile-Specific Interactions

Cua implements multi-touch gestures and mobile interactions through a transport-agnostic architecture that abstracts Android input commands into a high-level async API, supporting both local ADB and remote gRPC emulator connections.

The Cua framework (available at trycua/cua) provides a Python-based automation layer for Android devices that cleanly separates gesture intent from low-level event injection. Its mobile interface allows developers to execute complex multi-touch gestures—from simple taps to N-finger swipes—while the transport layer handles the device-specific implementation via Android Debug Bridge (ADB) or remote emulator protocols.

Transport-Agnostic Architecture for Mobile Automation

Cua’s mobile stack is organized into three distinct layers that isolate device communication from gesture logic:

• High-Level API – The Mobile class in libs/python/cua-sandbox/cua_sandbox/interfaces/mobile.py exposes async methods for taps, swipes, pinches, and hardware keys.
• Transport Abstraction – The Transport base class in libs/python/cua-sandbox/cua_sandbox/transport/base.py defines the contract for sending commands, with concrete implementations for ADB and gRPC.
• Multi-Touch Injection – Low-level sendevent sequences implementing Android’s MT Protocol B are generated in libs/python/cua-sandbox/cua_sandbox/transport/adb.py for local devices, and mirrored in libs/python/cua-sandbox/cua_sandbox/transport/grpc_emulator.py for remote cloud emulators.

The Mobile Class and High-Level Gesture API

The Mobile class serves as the primary interface for mobile-specific interactions, wrapping transport-specific details into intuitive async methods.

Single-Touch Operations

Standard single-pointer actions execute via adb shell input commands:

• tap(x, y) – Quick touch and release at screen coordinates.
• long_press(x, y, duration_ms) – Extended press with configurable hold time.
• swipe(x1, y1, x2, y2, duration_ms) – Linear drag operation.
• scroll_* and fling – Momentum-based scrolling utilities.

Before executing gestures, the class queries screen dimensions via wm size to translate logical pixel coordinates to the device’s native resolution.

Generic N-Finger Gestures

For complex multi-touch gestures, the gesture() method accepts an arbitrary number of finger paths and forwards a structured payload to the transport layer:

await mobile.gesture(
    (cx - 20, cy), (cx - 200, cy),   # finger 0: start → end

    (cx + 20, cy), (cx + 200, cy),   # finger 1: start → end

    duration_ms=400
)

The method validates path arguments, constructs per-finger coordinate pairs, and sends the "multitouch_gesture" action to the transport with screen_w, screen_h, and interpolation steps parameters.

Convenience Helpers for Pinch Gestures

pinch_in and pinch_out are thin wrappers around gesture() that calculate two-finger start and end points centered on (cx, cy):

await mobile.pinch_out(cx=540, cy=960, spread=250, duration_ms=500)

Low-Level Multi-Touch Implementation via ADB

When Mobile dispatches a "multitouch_gesture" action, ADBTransport._multitouch_gesture in libs/python/cua-sandbox/cua_sandbox/transport/adb.py handles the kernel-level event injection:

1. Root Access – Executes adb root to ensure sufficient privileges for /dev/input/event* access.
2. Input Node Discovery – Identifies the correct touch device by scanning for ABS_MT_POSITION_X (code 0x0035) capability reports.
3. Coordinate Mapping – Reads the device’s axis maximum (typically 32767) to create a px_to_raw scaling function.
4. MT Protocol B Sequence Generation – Constructs a chain of sendevent commands:
   • Assigns slots (ABS_MT_SLOT) and tracking IDs (ABS_MT_TRACKING_ID) for each finger.
   • Sets X/Y coordinates (ABS_MT_POSITION_X/Y) and pressure values.
   • Emits synchronization events (SYN_REPORT) and BTN_TOUCH = 1 for the initial contact.
   • Interpolates intermediate positions for smooth movement across the specified duration_ms.
   • Releases contacts by setting TID_NONE and BTN_TOUCH = 0, followed by a final sync.
5. Batch Execution – Joins all commands with && and executes via a single adb shell invocation.

This approach implements the MT Protocol B standard that the Android kernel interprets as simultaneous touch contacts, enabling true multi-touch simulation rather than sequential single-touch events.

Remote Emulator and gRPC Transport

For cloud-based testing environments, GrpcEmulatorTransport in libs/python/cua-sandbox/cua_sandbox/transport/grpc_emulator.py accepts the same gesture payload structure and forwards it via gRPC to a remote emulator server.

The server-side handler in libs/python/computer-server/computer_server/handlers/android.py receives the multitouch_gesture RPC and executes identical sendevent logic within the emulator’s ADB environment. This ensures that high-level API calls work transparently across both local USB-connected devices and remote cloud emulators without code changes.

Mobile-Specific Interactions Beyond Touch

Cua handles non-gesture mobile interactions through the same transport abstraction:

• Hardware Keys – home(), back(), recents(), power(), and volume controls map to input keyevent <code> shell commands.
• Text Input – enter() and backspace() methods send specific keyevent codes.
• System Actions – wake(), notifications(), and close_notifications() use either key events or direct service call invocations to manipulate the Android system UI.

All methods are asynchronous and share the transport layer, ensuring consistent behavior across ADB and remote back-ends.

Practical Implementation Examples

Basic Single-Touch Tap

from cua_sandbox.transport.adb import ADBTransport
from cua_sandbox.interfaces.mobile import Mobile

transport = ADBTransport(serial="emulator-5554")
await transport.connect()
mobile = Mobile(transport)

await mobile.tap(300, 600)          # tap at (300, 600) screen pixels

await transport.disconnect()

Two-Finger Pinch-Out (Zoom-In)

await mobile.pinch_out(cx=540, cy=960, spread=250, duration_ms=500)

The cx and cy parameters specify the pinch center, while spread determines the distance each finger travels from that center.

Arbitrary Three-Finger Swipe

await mobile.gesture(
    (100, 100), (100, 800),   # Finger 0

    (300, 100), (300, 800),   # Finger 1

    (500, 100), (500, 800),   # Finger 2

    duration_ms=800,
)

Remote Cloud Emulator Usage

from cua_sandbox.transport.grpc_emulator import GrpcEmulatorTransport

transport = GrpcEmulatorTransport(host="emu.example.com", token="…")
await transport.connect()
mobile = Mobile(transport)

await mobile.pinch_in(cx=540, cy=960, spread=200)
await transport.disconnect()

Summary

• Cua provides transport-agnostic mobile automation through the Mobile class, supporting both ADB and gRPC transports.
• Multi-touch gestures are handled via the gesture() method, which supports arbitrary N-finger paths with configurable duration and interpolation.
• ADBTransport._multitouch_gesture implements MT Protocol B sendevent sequences to inject true simultaneous touch events at the kernel level.
• Remote emulators receive identical gesture payloads via gRPC, with server-side execution matching local ADB behavior.
• All interactions—including hardware keys and system actions—are async and work uniformly across local and cloud Android targets.

Frequently Asked Questions

How does Cua distinguish between single-touch and multi-touch gestures?

Single-touch actions like tap() and swipe() use standard adb shell input commands, while multi-touch gestures invoke the gesture() method which constructs a complex payload containing multiple finger paths. The transport layer detects this payload type and routes it to _multitouch_gesture for low-level sendevent injection rather than simple shell input commands.

What Android permissions are required for Cua’s multi-touch injection?

The device must allow adb root access because the implementation writes directly to /dev/input/event* nodes using the sendevent command. This requires root privileges to bypass standard Android input stack restrictions and inject raw MT Protocol B events at the kernel driver level.

Can Cua handle gestures on emulators as well as physical devices?

Yes. The architecture supports both physical devices via ADBTransport and remote emulators via GrpcEmulatorTransport. The GrpcEmulatorTransport forwards the same gesture payload to a cloud server running computer_server/handlers/android.py, which executes identical sendevent logic inside the emulator’s environment, ensuring consistent behavior across deployment targets.

How does Cua calculate intermediate positions for smooth gestures?

When steps is set to 0 (auto-compute) or a specific value, the transport implementation interpolates between start and end coordinates for each finger across the duration_ms timeframe. It generates discrete sendevent commands for each intermediate position, creating smooth linear movement that mimics natural touch input rather than instantaneous teleportation between points.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}