# How Maka Handles Computer Use Backend Selection and the Display‑Snapshot Protocol

> Maka selects Computer Use backends with a pluggable system and processes display snapshots through a strict protocol. It protects host privacy by filtering executor output before it reaches the model.

- Repository: [The Apache Software Foundation/maka](https://github.com/apache/maka)
- Tags: internals
- Published: 2026-08-31

---

**Maka selects a Computer Use backend through a pluggable selector system and processes display snapshots via a strict protocol that protects host privacy by filtering all executor output before it reaches the model.**

Maka's Computer Use feature enables AI models to observe and interact with desktop applications. The architecture separates backend selection from snapshot handling, using well-defined interfaces defined in [`packages/runtime/src/computer-use-types.ts`](https://github.com/apache/maka/blob/main/packages/runtime/src/computer-use-types.ts) and implemented across the `packages/computer-use/src/` directory. Understanding this two-stage pipeline is essential for anyone deploying or debugging Maka's host integration capabilities.

## Computer Use Backend Selection Architecture

Maka's backend selection follows a **selector pattern** that instantiates concrete implementations of the `CuDispatchBackend` interface. The selector lives in [`packages/computer-use/src/select-backend.ts`](https://github.com/apache/maka/blob/main/packages/computer-use/src/select-backend.ts) and evaluates runtime conditions to choose between available backends.

### Available Backend Implementations

The selector currently recognizes two backends:

| Backend | Source File | Selection Condition |
|---------|-------------|---------------------|
| **cua-driver** | [`packages/computer-use/src/cua-driver-backend.ts`](https://github.com/apache/maka/blob/main/packages/computer-use/src/cua-driver-backend.ts) | `MAKA_CU_BACKEND="cua-driver"` environment variable set **and** binary present |
| **maka-cu** | [`packages/computer-use/src/maka-cu-backend.ts`](https://github.com/apache/maka/blob/main/packages/computer-use/src/maka-cu-backend.ts) | Selector requested for `"maka-cu"` (default) **and** executable can launch |

The **maka-cu backend** ships by default and serves as the production implementation. The **cua-driver backend** remains available for specialized deployments but requires explicit opt-in via environment variable.

### Backend Instantiation and Configuration

The selector constructs backends using `MakaCuBackendOptions`, which includes:

- `binaryPath` — Path to the executor binary
- `hostVersion` — Protocol version for compatibility checks
- `timeoutMs` — Operation timeout
- `onTrace` — Callback for diagnostic events
- `allowCompatibilityInputDispatch` — Feature flag for legacy input methods

If instantiation fails due to missing binaries, hash mismatches, or protocol version incompatibility, the selector throws `MakaCuHostRefusal` with error code `service_unavailable`. The runtime surfaces this as a **host-error sentence** defined in `HOST_ERROR_SENTENCE` (lines 100-121 of the backend file).

```typescript
import { createMakaCuBackend } from '@maka/computer-use';
import { selectBackend } from '@maka/computer-use/select-backend';

// Environment-variable selection pattern
const backend = selectBackend(process.env.MAKA_CU_BACKEND ?? 'maka-cu');

```

## The Display-Snapshot Protocol

Once a backend is active, every window observation produces a **snapshot** governed by the protocol in [`packages/runtime/src/computer-use-types.ts`](https://github.com/apache/maka/blob/main/packages/runtime/src/computer-use-types.ts). The maka-cu backend implements this protocol in [`packages/computer-use/src/maka-cu-backend.ts`](https://github.com/apache/maka/blob/main/packages/computer-use/src/maka-cu-backend.ts).

### Snapshot Identification and Storage

Each snapshot receives:

- A **UUID** (`snapshotId`) for session-scoped reference
- A **per-session map** of element tokens to digests, stored in `StoredSnapshot` (lines 23-33)

This mapping enables the model to reference UI elements across multiple turns without resending full observation data.

### Extended Element Metadata

The backend expands the shared `CuObservedElement` type with `MakaCuObservedElement` (lines 72-86), adding fields:

| Field | Purpose |
|-------|---------|
| `subrole` | Semantic sub-classification of element type |
| `placeholder` | Hint text for input fields |
| `actions` | Available interactions for this element |
| `focused` | Current keyboard focus state |

These fields are **not yet part of the public contract** but are validated and emitted through `type: 'observe'` trace events for debugging.

### Truncation and Partial Observation Handling

The executor may capture only a subset of the UI tree. The protocol communicates this through boolean flags on observation events (lines 94-106):

```typescript
// Truncation flags in observation events
truncatedElements: boolean      // Element list was cut off
truncatedDepth: boolean         // Tree depth limit reached
truncatedTextElements: number   // Count of text elements omitted

```

These flags travel exclusively through the `onTrace` callback. They represent the **only channel** for reporting partial observations to the model, which must then decide whether to request additional snapshots or proceed with available information.

### Host-Error Sentence Generation

All backend failures—protocol violations, executor crashes, timeouts, permission denials—map to fixed sentences in `HOST_ERROR_SENTENCE`. The backend constructs a `CaptureFailure` object with `messageIsAppTextFree: true`, guaranteeing that **no raw executor output reaches the model**.

```typescript
try {
  await cuBackend.runSemantic(action, abortSignal, runContext);
} catch (e) {
  if (e instanceof MakaCuHostRefusal) {
    // e.code: ComputerUseErrorCode
    // e.message: Sanitized, user-facing sentence
    console.error('Computer Use failed:', e.message);
  }
}

```

### Snapshot Lifecycle and Expiration

Snapshots expire based on negotiated service limits (`limits.snapshotTtlMs`). When expiration occurs, the backend records a `ForgottenReason`:

| Reason | Trigger |
|--------|---------|
| `expired` | TTL exceeded |
| `evicted` | Cache pressure |
| `spent` | Element already consumed |
| `superseded` | Newer snapshot available |

Each reason maps to a specific explanatory sentence from `FORGOTTEN_SENTENCE` (lines 174-182), allowing the model to choose between re-observation or alternative strategies.

## Two-Stage Pipeline Overview

The complete data flow for Computer Use backend selection and display-snapshot handling:

```

Model → Runtime → select-backend.ts → chosen backend 
  → maka-cu executor ↔ Host protocol 
  → Snapshot data → Runtime → Model

```

Because the backend is **host-owned**, all raw diagnostics flow through `onTrace` exclusively. The model receives only sanitized, structured observation data and fixed error sentences—preserving both privacy and predictable behavior.

## Key Implementation Files

| File | Responsibility |
|------|--------------|
| [`packages/computer-use/src/select-backend.ts`](https://github.com/apache/maka/blob/main/packages/computer-use/src/select-backend.ts) | Backend selection logic, `CuDispatchBackend` instantiation |
| [`packages/computer-use/src/maka-cu-backend.ts`](https://github.com/apache/maka/blob/main/packages/computer-use/src/maka-cu-backend.ts) | maka-cu protocol implementation, snapshot management, error translation |
| [`packages/runtime/src/computer-use-types.ts`](https://github.com/apache/maka/blob/main/packages/runtime/src/computer-use-types.ts) | Shared interfaces, snapshot types, protocol definitions |
| [`packages/runtime/src/computer-use-observation-text.ts`](https://github.com/apache/maka/blob/main/packages/runtime/src/computer-use-observation-text.ts) | Observation formatting for model consumption |
| [`packages/runtime/src/computer-use-tools.ts`](https://github.com/apache/maka/blob/main/packages/runtime/src/computer-use-tools.ts) | Tool wrapper routing calls to selected backend |

## Summary

- **Backend selection** is environment-driven: `MAKA_CU_BACKEND` chooses between cua-driver and maka-cu, with maka-cu as the secure default.
- **Snapshot protocol** enforces UUID-based identification, element token mapping, and truncation flag communication.
- **Privacy preservation** is architectural: `HOST_ERROR_SENTENCE` and `messageIsAppTextFree` ensure zero raw executor output reaches the model.
- **Expiration handling** provides granular `ForgottenReason` values so models can adapt to stale snapshot conditions.
- **Traceability** is complete via `onTrace` callbacks without compromising the model-facing interface.

## Frequently Asked Questions

### How do I force Maka to use a specific Computer Use backend?

Set the `MAKA_CU_BACKEND` environment variable to `"cua-driver"` or `"maka-cu"`. If unset, the selector defaults to `"maka-cu"` when the selector is enabled. The cua-driver backend requires both the environment variable and a present binary; otherwise, [`select-backend.ts`](https://github.com/apache/maka/blob/main/select-backend.ts) will reject instantiation with `service_unavailable`.

### What happens when a snapshot is truncated?

The backend sets `truncatedElements`, `truncatedDepth`, or `truncatedTextElements` flags on the observation event. These appear only in `onTrace` callbacks, not in the model-facing response. The model receives the partial element list and must infer truncation from context or request re-observation if critical elements are missing.

### Why does Maka use fixed error sentences instead of raw executor output?

The `messageIsAppTextFree: true` flag in `CaptureFailure` objects ensures that application text, system paths, and other host-sensitive data never reach the model. This prevents information leakage while still signaling failure modes through predictable, testable error codes mapped to `HOST_ERROR_SENTENCE` entries.

### How long do snapshots remain valid?

Snapshot lifetime is bound by `limits.snapshotTtlMs` from service negotiation, not a hardcoded constant. When expired, the backend returns a `FORGOTTEN_SENTENCE` with reason `expired`, `evicted`, `spent`, or `superseded`. Applications can tune this through backend options or request fresh observations when receiving expiration notices.