# How the Durable Agent Runner Works in OpenMAIC: Fault-Tolerant AI Execution Explained

> Discover how OpenMAICs durable agent runner ensures fault-tolerant AI execution by persisting interactions in PostgreSQL. Resume agents after crashes without data loss.

- Repository: [MAIC/OpenMAIC](https://github.com/THU-MAIC/OpenMAIC)
- Tags: deep-dive
- Published: 2026-08-30

---

**The durable agent runner in OpenMAIC persists every interaction—from user messages to tool results—as immutable events in PostgreSQL, enabling agents to resume after crashes without losing context or duplicating actions.**

The **durable agent runner** is the core execution engine of the OpenMAIC framework that transforms stateless LLM interactions into stateful, fault-tolerant conversations. Built on top of the Pi-Agent core, this system ensures that every turn is recorded as a durable event with strict ordering guarantees. By combining append-only storage with PostgreSQL's `LISTEN/NOTIFY` mechanism, OpenMAIC provides exactly-once execution semantics for long-running AI workflows.

## Core Architecture Components

The runner consists of six tightly integrated components that together form a resilient execution pipeline.

### buildAgent: Constructing the Runtime

The `buildAgent` function in [`lib/agent/runtime/build-agent.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/agent/runtime/build-agent.ts) constructs a Pi `Agent` instance with custom durable-aware hooks. It accepts a `streamFn`, a list of tools, and quota/allow-list configurations. Crucially, it installs a **terminal barrier** that pauses the agent when a length-limited turn finishes, preventing duplicate assistant messages while waiting for client acknowledgment.

### runNativeChild: The Execution Loop

Located in [`lib/agent/runtime/run-native-child.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/agent/runtime/run-native-child.ts), `runNativeChild` is the heart of the durable runner. It spawns a native child process that drives the agent through discrete **turns**. Each turn generates durable events (such as `user_message`, `assistant_message`, `tool_call`, and `tool_result`) with monotonically increasing `seq` integers. The runner maintains a PostgreSQL client that listens for database notifications via `LISTEN/NOTIFY`, waking immediately when new durable rows appear from concurrent processes.

### streamFn: Bridging Streams and Storage

The `streamFn` implementation in [`lib/agent/runtime/stream-fn.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/agent/runtime/stream-fn.ts) forwards the LLM's token stream to the client while simultaneously writing a **durable receipt** that marks the exact point where the LLM finished generating. This dual-write pattern ensures that the UI remains responsive while the durability log remains consistent.

### Tool Timeout and Error Handling

The [`lib/agent/runtime/tool-timeout.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/agent/runtime/tool-timeout.ts) module wraps every `AgentTool` with a global timeout. If a tool hangs, the runner writes a **durable error** row to the log and continues execution, guaranteeing that the durable log never stalls due to misbehaving external APIs.

### Quota Enforcement

The [`lib/agent/runtime/quota.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/agent/runtime/quota.ts) module provides a lightweight hook that aborts turns when users exceed predefined limits. Like other hooks, it contributes a durable receipt to the log, ensuring that quota violations are recorded immutably alongside the conversation history.

### Storage Layer

The `@openmaic/storage` package manages PostgreSQL tables including `agent_session`, `agent_message`, and `agent_material`. The runner performs upserts against these tables during execution and reads them during replay to reconstruct session state.

## How the Runner Guarantees Durability

OpenMAIC implements five specific mechanisms to ensure fault tolerance:

**Immutable event ordering**. Every durable event receives a monotonically increasing `seq` integer. The runner never mutates existing rows; it only appends new ones. This append-only design enables deterministic replay of any session state.

**Recovery markers**. When interrupted (e.g., during a server restart), the runner writes a recovery marker—a durable receipt—before halting. On restart, `runNativeChild` loads the last durable row and resumes execution immediately after the marker, ensuring exactly-once execution of tool calls as verified in [`tests/agent-runtime/runner-wakeup.test.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/tests/agent-runtime/runner-wakeup.test.ts).

**Listen/Notify wake-up**. The database triggers a PostgreSQL `NOTIFY` on every durable insert. The runner holds a persistent client connection listening for these notifications, enabling instant reaction to new rows written by other processes such as a concurrent UI.

**Terminal barrier**. When the LLM stops due to length limits, the terminal barrier blocks further input until the client acknowledges the receipt. This prevents race conditions that could generate duplicate assistant messages.

**Tool-call integrity**. Tool calls are wrapped with timeouts and receipt tracking that records the call ID and result. If a tool times out, the runner writes a durable error row and continues, ensuring the log remains consistent even during partial failures.

## Implementation Example: Starting a Durable Session

The following patterns demonstrate the three typical lifecycle phases of a durable session.

### 1. Building the Agent

```typescript
import { buildAgent } from '@/lib/agent/runtime/build-agent';
import { streamFn } from '@/lib/agent/runtime/stream-fn';
import { mySkillTools } from '@/my-skills';

const agent = buildAgent({
  streamFn,
  systemPrompt: 'You are a helpful teaching assistant.',
  tools: mySkillTools,
  allowedToolNames: new Set(['search', 'fetchUrl']),
  afterToolCall: async (ctx) => {
    return { metadata: { requestId: ctx.requestId } };
  },
});

```

### 2. Launching the Runner

```typescript
import { runNativeChild } from '@/lib/agent/runtime/run-native-child';

await runNativeChild({
  agent,
  sessionId: 'c1a2b3d4-5678-90ab-cdef-1234567890ab',
  abortSignal: controller.signal,
});

```

### 3. Recovering an Interrupted Session

```typescript
import { recoverSession } from '@/lib/server/agent-runtime/agent-driver-model';

const recoveredAgent = await recoverSession('c1a2b3d4-5678-90ab-cdef-1234567890ab');
await runNativeChild({ 
  agent: recoveredAgent, 
  sessionId: recoveredAgent.sessionId 
});

```

The **Create** phase initializes the first durable rows. The **Run** phase continuously appends rows while the LLM streams, using `LISTEN/NOTIFY` to synchronize with the UI. The **Recover** phase loads the last durable receipt and resumes without re-executing previous tool calls.

## Summary

- **The durable agent runner** persists every turn as an immutable event in PostgreSQL with monotonically increasing sequence numbers.
- **`runNativeChild`** in [`lib/agent/runtime/run-native-child.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/agent/runtime/run-native-child.ts) drives the execution loop and reacts to database notifications for real-time synchronization.
- **Recovery markers** enable exactly-once execution semantics after crashes, preventing duplicate tool calls or message loss.
- **Terminal barriers** and **tool timeouts** ensure that length limits and hanging external APIs cannot corrupt the durable log.
- The **`buildAgent`** factory in [`lib/agent/runtime/build-agent.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/agent/runtime/build-agent.ts) wires together streaming, quotas, and tool constraints with durability hooks.

## Frequently Asked Questions

### What happens if the server crashes during a tool call?

The runner writes a durable receipt immediately before executing a tool. If the server crashes, the recovery process in [`agent-driver-model.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/agent-driver-model.ts) reads the last receipt during restart and resumes execution from that point. If the tool had already executed, the receipt prevents re-execution; if it had not executed, the runner retries the call.

### How does the runner prevent duplicate messages when the LLM hits token limits?

The **terminal barrier** installed by `buildAgent` pauses the agent after a length-limited turn. The runner blocks further processing until the client acknowledges receipt of the partial message. This synchronization point ensures that the client and server agree on the conversation state before proceeding.

### Can multiple processes write to the same durable session simultaneously?

Yes. The runner uses PostgreSQL's `LISTEN/NOTIFY` mechanism to watch for new durable rows. When one process writes a `user_message`, the database notifies all listening runners. The `runNativeChild` implementation handles these notifications to wake sleeping processes and coordinate access to shared session state.

### Where are the durability guarantees tested?

The test suite in [`tests/agent-runtime/runner-wakeup.test.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/tests/agent-runtime/runner-wakeup.test.ts) verifies recovery behavior, while [`tests/agent-runtime/runner-event-order.test.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/tests/agent-runtime/runner-event-order.test.ts) validates immutable sequencing. Tool timeout behavior is covered in [`tests/lib/agent/runtime/tool-timeout.test.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/tests/lib/agent/runtime/tool-timeout.test.ts).