How the Durable Agent Runner Works in OpenMAIC: Fault-Tolerant AI Execution Explained

The durable agent runner in OpenMAIC persists every interaction—from user messages to tool results—as immutable events in PostgreSQL, enabling agents to resume after crashes without losing context or duplicating actions.

The durable agent runner is the core execution engine of the OpenMAIC framework that transforms stateless LLM interactions into stateful, fault-tolerant conversations. Built on top of the Pi-Agent core, this system ensures that every turn is recorded as a durable event with strict ordering guarantees. By combining append-only storage with PostgreSQL's LISTEN/NOTIFY mechanism, OpenMAIC provides exactly-once execution semantics for long-running AI workflows.

Core Architecture Components

The runner consists of six tightly integrated components that together form a resilient execution pipeline.

buildAgent: Constructing the Runtime

The buildAgent function in lib/agent/runtime/build-agent.ts constructs a Pi Agent instance with custom durable-aware hooks. It accepts a streamFn, a list of tools, and quota/allow-list configurations. Crucially, it installs a terminal barrier that pauses the agent when a length-limited turn finishes, preventing duplicate assistant messages while waiting for client acknowledgment.

runNativeChild: The Execution Loop

Located in lib/agent/runtime/run-native-child.ts, runNativeChild is the heart of the durable runner. It spawns a native child process that drives the agent through discrete turns. Each turn generates durable events (such as user_message, assistant_message, tool_call, and tool_result) with monotonically increasing seq integers. The runner maintains a PostgreSQL client that listens for database notifications via LISTEN/NOTIFY, waking immediately when new durable rows appear from concurrent processes.

streamFn: Bridging Streams and Storage

The streamFn implementation in lib/agent/runtime/stream-fn.ts forwards the LLM's token stream to the client while simultaneously writing a durable receipt that marks the exact point where the LLM finished generating. This dual-write pattern ensures that the UI remains responsive while the durability log remains consistent.

Tool Timeout and Error Handling

The lib/agent/runtime/tool-timeout.ts module wraps every AgentTool with a global timeout. If a tool hangs, the runner writes a durable error row to the log and continues execution, guaranteeing that the durable log never stalls due to misbehaving external APIs.

Quota Enforcement

The lib/agent/runtime/quota.ts module provides a lightweight hook that aborts turns when users exceed predefined limits. Like other hooks, it contributes a durable receipt to the log, ensuring that quota violations are recorded immutably alongside the conversation history.

Storage Layer

The @openmaic/storage package manages PostgreSQL tables including agent_session, agent_message, and agent_material. The runner performs upserts against these tables during execution and reads them during replay to reconstruct session state.

How the Runner Guarantees Durability

OpenMAIC implements five specific mechanisms to ensure fault tolerance:

Immutable event ordering. Every durable event receives a monotonically increasing seq integer. The runner never mutates existing rows; it only appends new ones. This append-only design enables deterministic replay of any session state.

Recovery markers. When interrupted (e.g., during a server restart), the runner writes a recovery marker—a durable receipt—before halting. On restart, runNativeChild loads the last durable row and resumes execution immediately after the marker, ensuring exactly-once execution of tool calls as verified in tests/agent-runtime/runner-wakeup.test.ts.

Listen/Notify wake-up. The database triggers a PostgreSQL NOTIFY on every durable insert. The runner holds a persistent client connection listening for these notifications, enabling instant reaction to new rows written by other processes such as a concurrent UI.

Terminal barrier. When the LLM stops due to length limits, the terminal barrier blocks further input until the client acknowledges the receipt. This prevents race conditions that could generate duplicate assistant messages.

Tool-call integrity. Tool calls are wrapped with timeouts and receipt tracking that records the call ID and result. If a tool times out, the runner writes a durable error row and continues, ensuring the log remains consistent even during partial failures.

Implementation Example: Starting a Durable Session

The following patterns demonstrate the three typical lifecycle phases of a durable session.

1. Building the Agent

import { buildAgent } from '@/lib/agent/runtime/build-agent';
import { streamFn } from '@/lib/agent/runtime/stream-fn';
import { mySkillTools } from '@/my-skills';

const agent = buildAgent({
  streamFn,
  systemPrompt: 'You are a helpful teaching assistant.',
  tools: mySkillTools,
  allowedToolNames: new Set(['search', 'fetchUrl']),
  afterToolCall: async (ctx) => {
    return { metadata: { requestId: ctx.requestId } };
  },
});

2. Launching the Runner

import { runNativeChild } from '@/lib/agent/runtime/run-native-child';

await runNativeChild({
  agent,
  sessionId: 'c1a2b3d4-5678-90ab-cdef-1234567890ab',
  abortSignal: controller.signal,
});

3. Recovering an Interrupted Session

import { recoverSession } from '@/lib/server/agent-runtime/agent-driver-model';

const recoveredAgent = await recoverSession('c1a2b3d4-5678-90ab-cdef-1234567890ab');
await runNativeChild({ 
  agent: recoveredAgent, 
  sessionId: recoveredAgent.sessionId 
});

The Create phase initializes the first durable rows. The Run phase continuously appends rows while the LLM streams, using LISTEN/NOTIFY to synchronize with the UI. The Recover phase loads the last durable receipt and resumes without re-executing previous tool calls.

Summary

• The durable agent runner persists every turn as an immutable event in PostgreSQL with monotonically increasing sequence numbers.
• runNativeChild in lib/agent/runtime/run-native-child.ts drives the execution loop and reacts to database notifications for real-time synchronization.
• Recovery markers enable exactly-once execution semantics after crashes, preventing duplicate tool calls or message loss.
• Terminal barriers and tool timeouts ensure that length limits and hanging external APIs cannot corrupt the durable log.
• The buildAgent factory in lib/agent/runtime/build-agent.ts wires together streaming, quotas, and tool constraints with durability hooks.

Frequently Asked Questions

What happens if the server crashes during a tool call?

The runner writes a durable receipt immediately before executing a tool. If the server crashes, the recovery process in agent-driver-model.ts reads the last receipt during restart and resumes execution from that point. If the tool had already executed, the receipt prevents re-execution; if it had not executed, the runner retries the call.

How does the runner prevent duplicate messages when the LLM hits token limits?

The terminal barrier installed by buildAgent pauses the agent after a length-limited turn. The runner blocks further processing until the client acknowledges receipt of the partial message. This synchronization point ensures that the client and server agree on the conversation state before proceeding.

Can multiple processes write to the same durable session simultaneously?

Yes. The runner uses PostgreSQL's LISTEN/NOTIFY mechanism to watch for new durable rows. When one process writes a user_message, the database notifies all listening runners. The runNativeChild implementation handles these notifications to wake sleeping processes and coordinate access to shared session state.

Where are the durability guarantees tested?

The test suite in tests/agent-runtime/runner-wakeup.test.ts verifies recovery behavior, while tests/agent-runtime/runner-event-order.test.ts validates immutable sequencing. Tool timeout behavior is covered in tests/lib/agent/runtime/tool-timeout.test.ts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}