# How the iii-sandbox Worker Provides Ephemeral VM Execution Environments from OCI Rootfs

> Learn how the iii-sandbox worker uses OCI rootfs to create ephemeral VM execution environments with libkrun microVMs for rapid, isolated testing.

- Repository: [iii/iii](https://github.com/iii-hq/iii)
- Tags: internals
- Published: 2026-05-28

---

**The iii-sandbox worker launches as an external child process, caches OCI rootfs images, and boots them inside libkrun micro-VMs with ephemeral overlays to provide isolated, short-lived execution environments.**

The **iii-sandbox** worker is an external worker implementation in the [iii-hq/iii](https://github.com/iii-hq/iii) repository that transforms standard OCI container images into lightweight, ephemeral virtual machines. Unlike traditional container runtimes, this worker extracts only the root filesystem, caches it locally, and executes commands inside hardware-virtualized micro-VMs through an overlay-based storage system.

## Architecture Overview

When a driver's [`config.yaml`](https://github.com/iii-hq/iii/blob/main/config.yaml) specifies the worker class `iii-sandbox`, the III engine resolves this via the known-external table in [`engine/src/workers/external.rs`](https://github.com/iii-hq/iii/blob/main/engine/src/workers/external.rs). The engine spawns the binary `iii-worker sandbox-daemon` as a child process, piping stdout and stderr to prevent terminal contamination. This daemon registers 14 distinct triggers—including `sandbox::create`, `sandbox::exec`, `sandbox::stop`, and filesystem operations—through the III SDK entry point in [`crates/iii-worker/src/sandbox_daemon/mod.rs`](https://github.com/iii-hq/iii/blob/main/crates/iii-worker/src/sandbox_daemon/mod.rs).

The architecture separates concerns across three layers:

- **Orchestration layer**: The engine manages worker lifecycle through `ExternalWorker::start_background_tasks`
- **Daemon layer**: The sandbox daemon handles OCI resolution, caching, and VM orchestration
- **Virtualization layer**: libkrun provides KVM-based micro-VMs with custom init systems

## The Lifecycle of an Ephemeral Sandbox

### Worker Launch and Registration

The lifecycle begins when `ExternalWorker::start_background_tasks` spawns `iii-worker sandbox-daemon`. The daemon immediately registers its API surface through functions like `register_sandbox_create`, `register_sandbox_exec`, and `register_sandbox_stop` in [`crates/iii-worker/src/sandbox_daemon/mod.rs`](https://github.com/iii-hq/iii/blob/main/crates/iii-worker/src/sandbox_daemon/mod.rs). This registration makes the sandbox triggers available to the III SDK for remote invocation.

### Image Resolution and Caching

When a client invokes `sandbox::create`, the daemon executes `handle_create` in [`crates/iii-worker/src/sandbox_daemon/create.rs`](https://github.com/iii-hq/iii/blob/main/crates/iii-worker/src/sandbox_daemon/create.rs). This process validates the request against an image allow-list, then resolves the image name to an OCI reference via the catalog module.

The rootfs handling follows a strict caching strategy:

1. Check for a cached unpacked rootfs at `~/.iii/cache/<slug>/`
2. If missing and `auto_install` is enabled, pull the OCI image using [`crates/iii-worker/src/sandbox_daemon/auto_install.rs`](https://github.com/iii-hq/iii/blob/main/crates/iii-worker/src/sandbox_daemon/auto_install.rs)
3. Extract and cache the filesystem for future reuse

This separation means the daemon never runs a full container engine—it only requires the extracted filesystem blob.

### VM Boot Process

After caching, `handle_create` constructs an ephemeral overlay directory structure (merged, upper, and work directories) via `overlay::OverlayLayout` in [`crates/iii-worker/src/sandbox_daemon/overlay.rs`](https://github.com/iii-hq/iii/blob/main/crates/iii-worker/src/sandbox_daemon/overlay.rs). The daemon then invokes `IiiWorkerLauncher::boot` from [`crates/iii-worker/src/sandbox_daemon/adapters.rs`](https://github.com/iii-hq/iii/blob/main/crates/iii-worker/src/sandbox_daemon/adapters.rs) to launch the micro-VM.

The boot sequence executes `iii-worker __vm-boot` (the same binary, re-executed) with flags specifying:

- Path to the cached rootfs (read-only)
- Overlay paths (writable upper layer)
- CPU and memory limits
- Unix sockets for control (`control.sock`) and command execution (`shell.sock`)

First-boot provisioning includes platform-specific setup: macOS code-signing and extraction of **libkrunfw** (the KVM hypervisor runtime) plus the `init.krun` binary via `ensure_libkrunfw` and `ensure_init_binary`. The launcher polls `shell.sock` until a successful connection confirms the VM is alive before returning the sandbox ID to the caller.

### Command Execution

Once booted, clients invoke `sandbox::exec` handled by `handle_exec` in [`crates/iii-worker/src/sandbox_daemon/exec.rs`](https://github.com/iii-hq/iii/blob/main/crates/iii-worker/src/sandbox_daemon/exec.rs). The daemon opens a **ShellRunner** that communicates with the running VM over `shell.sock` using the `iii-shell-client` protocol.

Commands execute inside the overlay's merged view, inheriting environment variables set at boot time plus any per-execution `env` overrides. Output streams back to the caller with a 1 MiB cap per stream (stdout and stderr separately).

### Filesystem Operations

The daemon provides ten filesystem triggers (`sandbox::fs::ls`, `write`, `read`, `rm`, etc.) implemented in [`crates/iii-worker/src/sandbox_daemon/fs.rs`](https://github.com/iii-hq/iii/blob/main/crates/iii-worker/src/sandbox_daemon/fs.rs). These manipulate the overlay's upper and work layers using standard POSIX calls through `iii-filesystem`. Because each sandbox maintains an isolated overlay, all mutations disappear when the sandbox stops, ensuring true ephemerality.

### Cleanup and Reaping

Resource cleanup operates through two mechanisms:

**Idle reaping**: A background task (`run_reaper_loop` in [`crates/iii-worker/src/sandbox_daemon/reaper.rs`](https://github.com/iii-hq/iii/blob/main/crates/iii-worker/src/sandbox_daemon/reaper.rs)) monitors `last_exec_at` timestamps. Sandboxes exceeding `idle_timeout_secs` (default 300 seconds) receive automatic termination and overlay deletion.

**Explicit stopping**: The `sandbox::stop` trigger (handled by `handle_stop` in [`crates/iii-worker/src/sandbox_daemon/stop.rs`](https://github.com/iii-hq/iii/blob/main/crates/iii-worker/src/sandbox_daemon/stop.rs)) sends SIGTERM, waits 200ms, then escalates to SIGKILL. The overlay directory is then deleted, freeing RAM and CPU while leaving the cached rootfs intact for reuse.

## Key Implementation Details

### OCI Rootfs Handling

According to the source code in [`crates/iii-worker/src/sandbox_daemon/auto_install.rs`](https://github.com/iii-hq/iii/blob/main/crates/iii-worker/src/sandbox_daemon/auto_install.rs), the auto-install logic pulls images only when missing from cache and validates them against the allow-list. This design ensures fast sandbox creation for subsequent uses of the same image while maintaining security boundaries.

### Overlay Filesystem

The **overlay::OverlayLayout** creates a unique directory triad for each sandbox:

- **upper/**: Captures all write operations
- **work/**: Required for overlay filesystem internals
- **merged/**: The unified view presented to the VM

Reads fallback to the immutable OCI rootfs when files are absent from the upper layer. When `sandbox::stop` executes, the daemon recursively deletes these three directories, instantly reverting the filesystem state.

### Resource Constraints

The daemon enforces hierarchical limits defined in the configuration:

- **Per-image caps**: `default_cpus`, `default_memory_mb`, and `per_image_caps` restrict individual sandboxes
- **Global limits**: `max_concurrent_sandboxes` prevents resource exhaustion
- **Error handling**: Requests exceeding limits return S-code `S400` before VM allocation occurs

## Code Examples

### TypeScript SDK

```typescript
import { registerWorker } from 'iii-sdk';

const iii = registerWorker('ws://127.0.0.1:49134');

// Boot a sandbox from the built-in "python" image
const { sandbox_id } = await iii.trigger({
  function_id: 'sandbox::create',
  payload: { image: 'python', cpus: 1, memory_mb: 512 },
  timeoutMs: 300_000,
});

// Run a command inside the VM
const out = await iii.trigger({
  function_id: 'sandbox::exec',
  payload: {
    sandbox_id,
    cmd: 'python3',
    args: ['-c', 'print(2 + 2)'],
  },
  timeoutMs: 35_000,
});
console.log(out.stdout); // → "4\n"

// Clean up
await iii.trigger({
  function_id: 'sandbox::stop',
  payload: { sandbox_id, wait: true },
});

```

### Python SDK

```python
from iii import register_worker

iii = register_worker('ws://127.0.0.1:49134')

# Create ephemeral VM

res = await iii.trigger({
    "function_id": "sandbox::create",
    "payload": {"image": "python", "cpus": 1, "memory_mb": 512},
    "timeout_ms": 300_000,
})
sandbox_id = res["sandbox_id"]

# Execute command

out = await iii.trigger({
    "function_id": "sandbox::exec",
    "payload": {"sandbox_id": sandbox_id,
                "cmd": "python3",
                "args": ["-c", "print(2 + 2)"]},
    "timeout_ms": 35_000,
})
print(out["stdout"])   # → "4\n"

# Terminate sandbox

await iii.trigger({
    "function_id": "sandbox::stop",
    "payload": {"sandbox_id": sandbox_id, "wait": True},
})

```

### CLI Shortcut

```bash
iii sandbox run python -- python3 -c 'print(2 + 2)'

# prints "4" and automatically stops the VM

```

## Summary

- **External process model**: The `iii-sandbox` worker runs as `iii-worker sandbox-daemon`, spawned by the engine via `ExternalWorker::start_background_tasks` in [`engine/src/workers/external.rs`](https://github.com/iii-hq/iii/blob/main/engine/src/workers/external.rs).
- **OCI to VM pipeline**: The daemon resolves OCI references, caches unpacked rootfs at `~/.iii/cache/<slug>/`, and boots them using `IiiWorkerLauncher::boot` with libkrun virtualization.
- **Ephemeral storage**: Each sandbox receives an isolated overlay (upper/work/merged directories) that is deleted on stop, while the cached rootfs remains for reuse.
- **Socket-based communication**: Commands flow through `shell.sock` using the `iii-shell-client` protocol, with output capped at 1 MiB per stream.
- **Automatic lifecycle management**: Idle sandboxes exceed `idle_timeout_secs` (default 300s) are reaped by `run_reaper_loop`, and explicit stops trigger SIGTERM → SIGKILL sequences.

## Frequently Asked Questions

### How does iii-sandbox differ from Docker or containerd?

Unlike Docker or containerd, the iii-sandbox worker does not run a container engine or manage daemon processes. According to the implementation in [`crates/iii-worker/src/sandbox_daemon/create.rs`](https://github.com/iii-hq/iii/blob/main/crates/iii-worker/src/sandbox_daemon/create.rs), it extracts only the OCI rootfs layer, caches it locally, and boots a libkrun micro-VM with that filesystem mounted read-only. Commands execute inside hardware-virtualized environments rather than kernel namespaces, providing stronger isolation boundaries while maintaining sub-second startup times for cached images.

### Where are the OCI images cached and how is storage managed?

The daemon stores unpacked rootfs directories under `~/.iii/cache/<slug>/` as implemented in [`crates/iii-worker/src/sandbox_daemon/auto_install.rs`](https://github.com/iii-hq/iii/blob/main/crates/iii-worker/src/sandbox_daemon/auto_install.rs). These cached layers persist across sandbox restarts for fast subsequent creation. However, each sandbox instance receives its own ephemeral overlay directory (created by `overlay::OverlayLayout` in [`crates/iii-worker/src/sandbox_daemon/overlay.rs`](https://github.com/iii-hq/iii/blob/main/crates/iii-worker/src/sandbox_daemon/overlay.rs)) that is deleted when the sandbox stops via `handle_stop` in [`crates/iii-worker/src/sandbox_daemon/stop.rs`](https://github.com/iii-hq/iii/blob/main/crates/iii-worker/src/sandbox_daemon/stop.rs). This design separates immutable image data from mutable sandbox state.

### What happens if a sandbox exceeds resource limits?

The daemon validates requests against `default_cpus`, `default_memory_mb`, `per_image_caps`, and `max_concurrent_sandboxes` before booting. Requests exceeding these limits immediately return S-code `S400` without spawning a VM, as defined in the resource enforcement logic. For sandboxes that exceed idle time, the `run_reaper_loop` in [`crates/iii-worker/src/sandbox_daemon/reaper.rs`](https://github.com/iii-hq/iii/blob/main/crates/iii-worker/src/sandbox_daemon/reaper.rs) automatically terminates them after `idle_timeout_secs` (default 300 seconds) of inactivity.

### Can I run custom OCI images or only built-in ones?

The sandbox supports custom OCI images when configured with an allow-list. The `handle_create` function in [`crates/iii-worker/src/sandbox_daemon/create.rs`](https://github.com/iii-hq/iii/blob/main/crates/iii-worker/src/sandbox_daemon/create.rs) validates image names against this list before resolution. If `auto_install` is enabled in the configuration, the daemon automatically pulls, extracts, and caches unauthorized but valid OCI references using the catalog module and [`auto_install.rs`](https://github.com/iii-hq/iii/blob/main/auto_install.rs) logic. Images must provide a standard Linux rootfs compatible with libkrun's virtualization layer.