# How the Native SDK Automation Server Enables AI Agent Testing and UI Snapshots

> Discover how vercel-labs/native's automation server streamlines AI agent testing with text-based UI snapshots and command queuing via a file protocol.

- Repository: [Vercel Labs/native](https://github.com/vercel-labs/native)
- Tags: how-to-guide
- Published: 2026-07-18

---

**The Native SDK automation server enables AI agent testing by exposing a drop-box file protocol that lets external scripts queue commands and read back deterministic, text-based UI snapshots of every window, widget, and runtime metric.**

The `vercel-labs/native` repository implements this server in Zig to bridge external test harnesses with a live Native app. By treating the filesystem as a lock-free message bus, the Native SDK automation server isolates the agent from the app process while still enabling full UI inspection and control. This architecture makes it safe for automated CI pipelines, large-scale AI-driven exploration, and snapshot-based regression testing.

## Drop-Box Directory and Protocol Handshake

All communication happens through a shared directory defined in `src/automation/protocol.zig`. The constant `default_dir` pins the drop-box to `.zig-cache/native-sdk-automation`, which both the CLI client and the running app watch.

```zig
pub const default_dir = ".zig-cache/native-sdk-automation"; // protocol.zig L3

```

Before accepting traffic, the server advertises its protocol version in `src/automation/server.zig`. When the runtime boots, it writes a `ready=true` line that embeds the current version—`7`—so the CLI can abort early on mismatch.

```zig
try writer.print("ready=true protocol={d} …\n", .{protocol.version}); // server.zig L40-L46

```

This handshake guarantees that the AI agent and the app speak the same contract before any commands or snapshots are exchanged.

You can initialise the server inside the native app with a single call:

```zig
const Server = @import("src/automation/server.zig").Server;

var server = Server.init(std.testing.io,
    ".zig-cache/native-sdk-automation", // directory
    "native-sdk");                       // title shown in logs

```

## Publishing UI Snapshots and Test Artifacts

When the runtime requests a publish cycle, the server writes three artifacts into the drop-box. Each write uses a `std.Io.Writer.Allocating` instance with pre-allocated capacity—**16 KB** for snapshots and **1 KB** for window metadata—followed by a single atomic `writePath` call.

- **[`snapshot.txt`](https://github.com/vercel-labs/native/blob/main/snapshot.txt)** — a full UI dump produced by `snapshot.writeText` starting at line 72 of `src/automation/snapshot.zig`.
- **[`accessibility.txt`](https://github.com/vercel-labs/native/blob/main/accessibility.txt)** — an assistive-technology tree produced by `snapshot.writeA11yText` starting at line 2 of `src/automation/snapshot.zig`.
- **[`bridge-response.txt`](https://github.com/vercel-labs/native/blob/main/bridge-response.txt)** / **[`provenance.txt`](https://github.com/vercel-labs/native/blob/main/provenance.txt)** — one-off responses handled by `publishBridgeResponse` and `publishProvenanceResponse` in `server.zig` lines 61-78.

```zig
var writer = try std.Io.Writer.Allocating.initCapacity(...);
try snapshot.writeText(input_value, &writer.writer);
try writePath(self.io, self.path("snapshot.txt", &path_buffer), writer.written());

```

A typical publish call looks like this:

```zig
var win = Window{
    .title = "Demo",
    .bounds = geometry.RectF.init(0, 0, 800, 600),
};
var widget = snapshot.Widget{
    .window_id = 1,
    .view_label = "main",
    .id = 100,
    .role = "button",
    .name = "ClickMe",
    .bounds = geometry.RectF.init(100, 100, 120, 40),
    .focused = true,
    .enabled = true,
};

try server.publish(.{
    .windows = &[_]snapshot.Window{win},
    .widgets = &[_]snapshot.Widget{widget},
});

```

For visual regression, the agent can issue a `screenshot` command. The runtime then calls `publishScreenshot` in `server.zig` (lines 80-97), which writes a temporary PNG and renames it atomically so no poller ever reads a truncated image.

## Command Queue: FIFO Dispatch and Crash Recovery

The server consumes work from a flat queue of `command-<seq>.txt` files dropped by external agents. Sequence numbers are generated by `protocol.queueFileName` in `protocol.zig` lines 57-60.

```zig
const name = try protocol.queueFileName(sequence, &name_buffer);

```

Dispatch is strictly **FIFO**. The helper `queueHeadName` in `server.zig` (lines 65-84) scans the directory for the smallest sequence number and returns the oldest entry.

```zig
fn queueHeadName(self: Server, name_buffer: []u8) ?[]const u8 { … } // server.zig L65-L84

```

The runtime calls `takeCommand` once per frame. This routine reads the head entry, deletes it immediately—the ACK—and returns a parsed `protocol.Command`. If the file is still being written and lacks a trailing newline, `takeCommand` defers processing and re-queues the watcher.

```zig
pub fn takeCommand(self: Server, buffer: []u8) !?protocol.Command { … } // server.zig L111-L142

```

Inside the app loop, consumption looks like this:

```zig
var read_buf: [256]u8 = undefined;
if (try server.takeCommand(&read_buf)) |command| {
    // Dispatch based on command.action …
    switch (command.action) {
        .widget_click => |*c| std.debug.print("Clicked widget: {s}\n", .{c.value}),
        else => {},
    }
}

```

To handle crashed writers, the server runs `reapAbandonedEntry` in `server.zig` (lines 86-95). Any incomplete file older than `abandoned_entry_reap_ns` is removed automatically. The protocol also caps total queued commands at **8** via `max_queued_commands` in `protocol.zig` lines 49-50, forcing writers to retry rather than allowing unbounded growth.

```zig
pub const max_queued_commands: usize = 8; // protocol.zig L49-L50

```

## AI Agent Testing Loop: Drive, Wait, and Observe

An AI agent interacts with the app through a simple four-phase loop. This decouples the agent from the process memory, making the Native SDK automation server ideal for untrusted or large-scale AI agents.

### Injecting Commands

The agent composes a command line with `protocol.commandLine` and writes it to a new `command-<seq>.txt` file.

```zig
var cmd_buf: [128]u8 = undefined;
const cmd = try protocol.commandLine("widget-click", "main 100", &cmd_buf);
try writePath(std.testing.io,
    server.path("command-1.txt", &path_buf), cmd);

```

### Waiting for Runtime Processing

After dropping the file, the agent polls `hasPendingCommand` in `server.zig` (lines 44-48) or watches the directory. The entry disappears only after the runtime acknowledges it.

```zig
while (server.hasPendingCommand()) std.time.sleep(50_000_000); // 50 ms

```

### Reading Snapshots for Verification

Once the command is consumed, the agent reads [`snapshot.txt`](https://github.com/vercel-labs/native/blob/main/snapshot.txt) or [`accessibility.txt`](https://github.com/vercel-labs/native/blob/main/accessibility.txt) to verify state, collect widget IDs, or feed the hierarchy into its own model.

```zig
var snap_buf: [64*1024]u8 = undefined;
const snap_text = try readPath(std.testing.io,
    server.path("snapshot.txt", &path_buf), &snap_buf);
// `snap_text` now contains lines like:
//   window @w1 "Demo" …
//   widget @w1/main#100 role=button name="ClickMe" …

```

### Capturing Screenshots

When the agent needs pixel-level verification, it queues a `screenshot` command. The resulting PNG is delivered atomically by `publishScreenshot`, guaranteeing that external readers never see a half-written image.

```zig
pub fn publishScreenshot(self: Server, view_label: []const u8, png_bytes: []const u8) !void { … } // server.zig L80-L97

```

## Summary

- The **Native SDK automation server** in `vercel-labs/native` uses a filesystem drop-box at `.zig-cache/native-sdk-automation` to decouple AI agents from the app process.
- **Protocol version 7** enforces compatibility before any commands are exchanged.
- The server publishes deterministic, text-based snapshots via `snapshot.writeText` and `snapshot.writeA11yText`, plus atomic screenshots via `publishScreenshot`.
- A **FIFO command queue** capped at **8** entries and protected by `reapAbandonedEntry` ensures safe, recoverable dispatch even when writers crash.
- AI agents drive the UI by writing `command-<seq>.txt` files, waiting on `hasPendingCommand`, and reading back structured UI state for verification or model training.

## Frequently Asked Questions

### How does the Native SDK automation server guarantee snapshot consistency?

The server never writes snapshots in place. Instead, it uses a `std.Io.Writer.Allocating` buffer to assemble the full text, then commits the file with a single atomic `writePath` call. Screenshot binaries follow the same pattern: a temp file is written and renamed only when complete. This ensures that AI agents always read valid, complete artifacts.

### What happens if an AI agent crashes while writing a command?

Incomplete command files are tracked by age. If a file sits unfinished longer than the `abandoned_entry_reap_ns` timeout, the server's `reapAbandonedEntry` routine—implemented in `src/automation/server.zig` lines 86-95—deletes it automatically. The queue also hard-limits outstanding commands to `max_queued_commands = 8`, so a crashed agent cannot exhaust server resources.

### Can multiple AI agents queue commands at the same time?

Yes, because the queue relies on unique sequence numbers generated by `protocol.queueFileName` in `src/automation/protocol.zig`. However, the runtime processes only one command per frame via `takeCommand`, and the oldest entry wins. If the queue hits its capacity of 8, additional writers must retry, which naturally serializes load.

### How does an AI agent know when a command has been executed?

The agent polls `hasPendingCommand` or watches the drop-box directory. A command is considered acknowledged only when `takeCommand` deletes its `command-<seq>.txt` file. Once the file disappears, the agent can safely read [`snapshot.txt`](https://github.com/vercel-labs/native/blob/main/snapshot.txt) or [`accessibility.txt`](https://github.com/vercel-labs/native/blob/main/accessibility.txt) to observe the updated UI state.