How the OpenMAIC Agent-Runtime Handles Leases and Crash Resumes with Server-Backed Durable Sessions

The server-backed durable session runner in OpenMAIC's lib/agent-runtime uses PostgreSQL leases with periodic heartbeats for exclusive access, detects lease loss via heartbeat failures and write rejections, and recovers from crashes by replaying persisted transcripts through planResume to rebuild a runnable conversation state.

The OpenMAIC project provides a robust agent-runtime system designed for production deployments where LLM-driven conversations must survive process restarts, worker crashes, and concurrent worker contention. This article examines how the durable session runner implements lease-based coordination and automatic crash recovery in lib/server/agent-runtime/runner.ts and its supporting modules.

---

Lease Acquisition and Validation

Every session execution begins with exclusive lease ownership. When a worker calls runSession with a ClaimedAgentSession, the claim contains two critical identifiers: workerId and the current attempt number. The runner validates this ownership before proceeding.

The leaseMatches utility enforces strict identity checks:

function leaseMatches(
  session: AgentSessionMeta | null,
  workerId: string,
  attempt: number,
): boolean {
  return session?.lease?.workerId === workerId && session.attempt === attempt;
}

A false return immediately triggers lease-lost handling, preventing split-brain scenarios where multiple workers might concurrently modify session state. This validation runs at key transition points throughout the session lifecycle.

---

Heartbeat Mechanism for Lease Refresh

Leases expire automatically without periodic renewal. The runner maintains a heartbeatTimer that fires every config.heartbeatIntervalMs (default approximately 5 seconds) to extend the lease in PostgreSQL.

const heartbeatTimer = setInterval(() => {
  store
    .heartbeat(id, WORKER_ID)
    .then((held) => {
      if (!held && !leaseLost) {
        log.warn(`session ${id}: lease lost; aborting local run`);
        markLeaseLost();
      }
    })
    .catch((error) => log.warn(`session ${id}: heartbeat failed`, error));
}, config.heartbeatIntervalMs);

The store.heartbeat method returns a boolean indicating whether this worker still holds the lease. If another worker claims the session or the lease expires due to missed heartbeats, the method returns false, triggering immediate local abort via markLeaseLost() and abort.abort().

---

Detecting Lease Loss During Durable Writes

Beyond heartbeats, lease loss surfaces during write operations. The runner queues all durable modifications through an enqueue system that wraps store calls. Critical path writes use writeRequiredSessionEntry, which interprets null responses as lease revocation:

enqueue(async () => {
  const seq = await store.appendRunEvent(...);
  if (seq === null) {
    markLeaseLost();          // lease no longer held
    return;
  }
  …
});

The helper isLeaseLostError traverses error chains to detect AgentSessionLeaseLostError instances, ensuring consistent handling across sync and async boundaries. This dual-detection strategy—proactive heartbeats and reactive write validation—eliminates race windows where a worker might perform stale operations.

---

Crash Detection and Transcript Recovery

When a worker terminates unexpectedly—after persisting partial progress but before graceful completion—the next acquiring worker must reconstruct a runnable conversation state. This recovery flows through lib/server/agent-runtime/resume.ts.

Loading Historical State

The runner retrieves the complete entry-tree history via loadEntryHistory and extracts recovery.messages, the durable transcript of all prior interactions.

The planResume Classification Engine

The core recovery logic resides in planResume (lines 94–126 of resume.ts). It sanitizes the transcript tail and classifies recovery actions:

Tail State	ResumeAction.kind	Behavior
Empty message array	'start'	Fresh session initiation with original prompt
Ends with user message or toolResult containing successful ask_user/create_skill	'already-complete'	Session paused awaiting external input; do not auto-respond
Ends with assistant message having pending tool calls	'continue' + repairedToolCalls	Resume mid-execution, re-issuing dangling calls
Ends with completed assistant turn (no pending calls)	'already-complete'	Terminal state; finalize as successful

The algorithm strips incomplete assistant frames—messages that are empty, aborted, or length-truncated—preventing the LLM from receiving malformed context.

---

Re-Issuing Dangling Tool Calls

For 'continue' cases, the runner invokes repairOrphanedToolCalls from lib/server/agent-runtime/tool-call-integrity.ts. This utility injects synthetic "interrupted" tool results for each dangling call ID identified by planResume:

const contextRepair = repairOrphanedToolCalls(plannedMessages);
const repairedToolCallIds = [
  ...new Set([
    ...(plan.kind === 'continue' ? plan.repairedToolCalls : []),
    ...contextRepair.repairedToolCalls,
  ]),
];

The Pi SDK (agent.continue()) perceives these as failed tool executions, prompting automatic retry. Tool implementations must be idempotent to safely handle this at-least-once execution guarantee.

---

Finalization and Re-Queueing Logic

Session Completion

Terminal events trigger store.finishSession. If lease loss occurred during the run, the commit aborts silently; the next worker will repeat the full resume flow. This ensures exactly-once finalization semantics modulo lease ownership.

Undelivered Message Handling

When sessions terminate without delivering all pending user messages, requeueIfUndelivered (via planUndeliveredRequeue) evaluates whether to reset the attempt counter for backoff purposes or immediate retry based on divergence between logged messages and last confirmed delivery sequence.

---

Practical Examples

Simulating Lease Loss

Test lease detection without actual crashes:

// Active runner holds sessionMeta with valid lease
const runnerHandle = await startRunner(sessionMeta);

// Force external lease acquisition
await store.forceLeaseSteal(sessionMeta.id, 'other-worker-id');

// Heartbeat detects loss within two intervals
await new Promise((resolve) => setTimeout(resolve, config.heartbeatIntervalMs * 2));
await runnerHandle.stop(); // Graceful cleanup

Triggering Crash Recovery Manually

// 1. Acquire session ownership
const meta = await claimSession('my-session-id');

// 2. Persist partial tool execution without completion
await store.appendRunEvent(meta.id, meta.workerId, {
  ts: Date.now(),
  attempt: meta.attempt,
  type: 'tool_execution_start',
  data: { toolName: 'generate_image', toolCallId: 't-42' },
});
// Deliberately omit tool_execution_end/toolResult

// 3. Release lease to simulate worker death
await store.releaseLease(meta.id, meta.workerId);

// 4. New worker automatically resumes and re-issues generate_image
await runSession(newRunContext, await claimSession(meta.id));

Detecting ask_user Pause States

const transcript = [
  { role: 'user', content: 'What do you think?' },
  { role: 'assistant', content: [{ type: 'toolCall', id: 'tc-1', name: 'ask_user' }] },
  { role: 'toolResult', toolName: 'ask_user', toolCallId: 'tc-1', isError: false },
];

const action = planResume(transcript);
console.log(action.kind); // 'already-complete' — awaiting human response

---

Key Implementation Files

File	Responsibility
lib/server/agent-runtime/runner.ts	Core state machine: lease claims, heartbeat coordination, event loop, lease-loss detection, LLM interaction
lib/server/agent-runtime/resume.ts	Post-crash transcript analysis, tail classification, planResume implementation
lib/server/agent-runtime/store.ts	PostgreSQL abstraction: claimSession, heartbeat, releaseLease, append operations
lib/server/agent-runtime/tool-call-integrity.ts	Synthetic result injection for orphaned tool calls
lib/server/agent-runtime/event-notify-bus.ts	NOTIFY-based subscription for external event wakeups during lease hold

---

Summary

• Lease coordination combines PostgreSQL-backed claims with 5-second heartbeat refreshes and write-time validation via leaseMatches.

• Crash recovery rebuilds runnable state from durable history using planResume, which classifies session tails into start, continue, or already-complete actions.

• Tool-call integrity is preserved through synthetic interruption results, enabling idempotent retry of dangling operations.

• Concurrent safety is guaranteed by immediate abort on any lease-loss detection, whether from heartbeat failure or rejected durable writes.

---

Frequently Asked Questions

How does the runner prevent two workers from processing the same session simultaneously?

The runner acquires an exclusive lease through claimSession in lib/server/agent-runtime/store.ts, validates ownership via leaseMatches on every critical operation, and maintains continuous heartbeat renewal. Any competing claim revokes the prior lease, causing heartbeat failures that trigger immediate local abort.

What happens if a worker crashes mid-tool-execution?

The next acquiring worker loads the persisted transcript through loadEntryHistory, calls planResume to identify dangling tool calls, and uses repairOrphanedToolCalls to inject synthetic failure results. The LLM then automatically retries these calls with full context intact.

Why must tools be idempotent in this system?

Because crash recovery may re-issue tool calls that partially executed before the original worker died. The repairOrphanedToolCalls mechanism treats all dangling calls as interrupted, causing automatic retry. Idempotent implementations ensure correct behavior despite multiple execution attempts.

How does the runner distinguish between a fresh session and a resumed one after crash?

planResume analyzes the transcript tail: an empty history yields 'start'; presence of ask_user or create_skill success results yields 'already-complete'; incomplete assistant turns with pending tool calls yield 'continue'. This classification drives subsequent runner behavior without external state.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}