How Maka Handles Computer Use Backend Selection and the Display‑Snapshot Protocol

Maka selects a Computer Use backend through a pluggable selector system and processes display snapshots via a strict protocol that protects host privacy by filtering all executor output before it reaches the model.

Maka's Computer Use feature enables AI models to observe and interact with desktop applications. The architecture separates backend selection from snapshot handling, using well-defined interfaces defined in packages/runtime/src/computer-use-types.ts and implemented across the packages/computer-use/src/ directory. Understanding this two-stage pipeline is essential for anyone deploying or debugging Maka's host integration capabilities.

Computer Use Backend Selection Architecture

Maka's backend selection follows a selector pattern that instantiates concrete implementations of the CuDispatchBackend interface. The selector lives in packages/computer-use/src/select-backend.ts and evaluates runtime conditions to choose between available backends.

Available Backend Implementations

The selector currently recognizes two backends:

Backend	Source File	Selection Condition
cua-driver	packages/computer-use/src/cua-driver-backend.ts	MAKA_CU_BACKEND="cua-driver" environment variable set and binary present
maka-cu	packages/computer-use/src/maka-cu-backend.ts	Selector requested for "maka-cu" (default) and executable can launch

The maka-cu backend ships by default and serves as the production implementation. The cua-driver backend remains available for specialized deployments but requires explicit opt-in via environment variable.

Backend Instantiation and Configuration

The selector constructs backends using MakaCuBackendOptions, which includes:

• binaryPath — Path to the executor binary
• hostVersion — Protocol version for compatibility checks
• timeoutMs — Operation timeout
• onTrace — Callback for diagnostic events
• allowCompatibilityInputDispatch — Feature flag for legacy input methods

If instantiation fails due to missing binaries, hash mismatches, or protocol version incompatibility, the selector throws MakaCuHostRefusal with error code service_unavailable. The runtime surfaces this as a host-error sentence defined in HOST_ERROR_SENTENCE (lines 100-121 of the backend file).

import { createMakaCuBackend } from '@maka/computer-use';
import { selectBackend } from '@maka/computer-use/select-backend';

// Environment-variable selection pattern
const backend = selectBackend(process.env.MAKA_CU_BACKEND ?? 'maka-cu');

The Display-Snapshot Protocol

Once a backend is active, every window observation produces a snapshot governed by the protocol in packages/runtime/src/computer-use-types.ts. The maka-cu backend implements this protocol in packages/computer-use/src/maka-cu-backend.ts.

Snapshot Identification and Storage

Each snapshot receives:

• A UUID (snapshotId) for session-scoped reference
• A per-session map of element tokens to digests, stored in StoredSnapshot (lines 23-33)

This mapping enables the model to reference UI elements across multiple turns without resending full observation data.

Extended Element Metadata

The backend expands the shared CuObservedElement type with MakaCuObservedElement (lines 72-86), adding fields:

Field	Purpose
subrole	Semantic sub-classification of element type
placeholder	Hint text for input fields
actions	Available interactions for this element
focused	Current keyboard focus state

These fields are not yet part of the public contract but are validated and emitted through type: 'observe' trace events for debugging.

Truncation and Partial Observation Handling

The executor may capture only a subset of the UI tree. The protocol communicates this through boolean flags on observation events (lines 94-106):

// Truncation flags in observation events
truncatedElements: boolean      // Element list was cut off
truncatedDepth: boolean         // Tree depth limit reached
truncatedTextElements: number   // Count of text elements omitted

These flags travel exclusively through the onTrace callback. They represent the only channel for reporting partial observations to the model, which must then decide whether to request additional snapshots or proceed with available information.

Host-Error Sentence Generation

All backend failures—protocol violations, executor crashes, timeouts, permission denials—map to fixed sentences in HOST_ERROR_SENTENCE. The backend constructs a CaptureFailure object with messageIsAppTextFree: true, guaranteeing that no raw executor output reaches the model.

try {
  await cuBackend.runSemantic(action, abortSignal, runContext);
} catch (e) {
  if (e instanceof MakaCuHostRefusal) {
    // e.code: ComputerUseErrorCode
    // e.message: Sanitized, user-facing sentence
    console.error('Computer Use failed:', e.message);
  }
}

Snapshot Lifecycle and Expiration

Snapshots expire based on negotiated service limits (limits.snapshotTtlMs). When expiration occurs, the backend records a ForgottenReason:

Reason	Trigger
expired	TTL exceeded
evicted	Cache pressure
spent	Element already consumed
superseded	Newer snapshot available

Each reason maps to a specific explanatory sentence from FORGOTTEN_SENTENCE (lines 174-182), allowing the model to choose between re-observation or alternative strategies.

Two-Stage Pipeline Overview

The complete data flow for Computer Use backend selection and display-snapshot handling:

Model → Runtime → select-backend.ts → chosen backend 
  → maka-cu executor ↔ Host protocol 
  → Snapshot data → Runtime → Model

Because the backend is host-owned, all raw diagnostics flow through onTrace exclusively. The model receives only sanitized, structured observation data and fixed error sentences—preserving both privacy and predictable behavior.

Key Implementation Files

File	Responsibility
packages/computer-use/src/select-backend.ts	Backend selection logic, CuDispatchBackend instantiation
packages/computer-use/src/maka-cu-backend.ts	maka-cu protocol implementation, snapshot management, error translation
packages/runtime/src/computer-use-types.ts	Shared interfaces, snapshot types, protocol definitions
packages/runtime/src/computer-use-observation-text.ts	Observation formatting for model consumption
packages/runtime/src/computer-use-tools.ts	Tool wrapper routing calls to selected backend

Summary

• Backend selection is environment-driven: MAKA_CU_BACKEND chooses between cua-driver and maka-cu, with maka-cu as the secure default.
• Snapshot protocol enforces UUID-based identification, element token mapping, and truncation flag communication.
• Privacy preservation is architectural: HOST_ERROR_SENTENCE and messageIsAppTextFree ensure zero raw executor output reaches the model.
• Expiration handling provides granular ForgottenReason values so models can adapt to stale snapshot conditions.
• Traceability is complete via onTrace callbacks without compromising the model-facing interface.

Frequently Asked Questions

How do I force Maka to use a specific Computer Use backend?

Set the MAKA_CU_BACKEND environment variable to "cua-driver" or "maka-cu". If unset, the selector defaults to "maka-cu" when the selector is enabled. The cua-driver backend requires both the environment variable and a present binary; otherwise, select-backend.ts will reject instantiation with service_unavailable.

What happens when a snapshot is truncated?

The backend sets truncatedElements, truncatedDepth, or truncatedTextElements flags on the observation event. These appear only in onTrace callbacks, not in the model-facing response. The model receives the partial element list and must infer truncation from context or request re-observation if critical elements are missing.

Why does Maka use fixed error sentences instead of raw executor output?

The messageIsAppTextFree: true flag in CaptureFailure objects ensures that application text, system paths, and other host-sensitive data never reach the model. This prevents information leakage while still signaling failure modes through predictable, testable error codes mapped to HOST_ERROR_SENTENCE entries.

How long do snapshots remain valid?

Snapshot lifetime is bound by limits.snapshotTtlMs from service negotiation, not a hardcoded constant. When expired, the backend returns a FORGOTTEN_SENTENCE with reason expired, evicted, spent, or superseded. Applications can tune this through backend options or request fresh observations when receiving expiration notices.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}