# How the OpenMAIC Agent-Runtime Handles Leases and Crash Resumes with Server-Backed Durable Sessions

> Discover how OpenMAIC's server-backed durable session runner in lib agent-runtime manages PostgreSQL leases, detects failures, and resumes conversations after crashes via transcript replays for reliable agent operation.

- Repository: [MAIC/OpenMAIC](https://github.com/THU-MAIC/OpenMAIC)
- Tags: internals
- Published: 2026-09-02

---

**The server-backed durable session runner in OpenMAIC's `lib/agent-runtime` uses PostgreSQL leases with periodic heartbeats for exclusive access, detects lease loss via heartbeat failures and write rejections, and recovers from crashes by replaying persisted transcripts through `planResume` to rebuild a runnable conversation state.**

The OpenMAIC project provides a robust agent-runtime system designed for production deployments where LLM-driven conversations must survive process restarts, worker crashes, and concurrent worker contention. This article examines how the durable session runner implements **lease-based coordination** and **automatic crash recovery** in [`lib/server/agent-runtime/runner.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/server/agent-runtime/runner.ts) and its supporting modules.

---

## Lease Acquisition and Validation

Every session execution begins with **exclusive lease ownership**. When a worker calls `runSession` with a `ClaimedAgentSession`, the claim contains two critical identifiers: `workerId` and the current `attempt` number. The runner validates this ownership before proceeding.

The `leaseMatches` utility enforces strict identity checks:

```typescript
function leaseMatches(
  session: AgentSessionMeta | null,
  workerId: string,
  attempt: number,
): boolean {
  return session?.lease?.workerId === workerId && session.attempt === attempt;
}

```

A `false` return immediately triggers **lease-lost** handling, preventing split-brain scenarios where multiple workers might concurrently modify session state. This validation runs at key transition points throughout the session lifecycle.

---

## Heartbeat Mechanism for Lease Refresh

Leases expire automatically without periodic renewal. The runner maintains a **`heartbeatTimer`** that fires every `config.heartbeatIntervalMs` (default approximately 5 seconds) to extend the lease in PostgreSQL.

```typescript
const heartbeatTimer = setInterval(() => {
  store
    .heartbeat(id, WORKER_ID)
    .then((held) => {
      if (!held && !leaseLost) {
        log.warn(`session ${id}: lease lost; aborting local run`);
        markLeaseLost();
      }
    })
    .catch((error) => log.warn(`session ${id}: heartbeat failed`, error));
}, config.heartbeatIntervalMs);

```

The `store.heartbeat` method returns a boolean indicating **whether this worker still holds the lease**. If another worker claims the session or the lease expires due to missed heartbeats, the method returns `false`, triggering immediate local abort via `markLeaseLost()` and `abort.abort()`.

---

## Detecting Lease Loss During Durable Writes

Beyond heartbeats, lease loss surfaces during **write operations**. The runner queues all durable modifications through an `enqueue` system that wraps store calls. Critical path writes use `writeRequiredSessionEntry`, which interprets `null` responses as lease revocation:

```typescript
enqueue(async () => {
  const seq = await store.appendRunEvent(...);
  if (seq === null) {
    markLeaseLost();          // lease no longer held
    return;
  }
  …
});

```

The helper `isLeaseLostError` traverses error chains to detect `AgentSessionLeaseLostError` instances, ensuring consistent handling across sync and async boundaries. This dual-detection strategy—**proactive heartbeats and reactive write validation**—eliminates race windows where a worker might perform stale operations.

---

## Crash Detection and Transcript Recovery

When a worker terminates unexpectedly—after persisting partial progress but before graceful completion—the next acquiring worker must **reconstruct a runnable conversation state**. This recovery flows through **[`lib/server/agent-runtime/resume.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/server/agent-runtime/resume.ts)**.

### Loading Historical State

The runner retrieves the complete entry-tree history via `loadEntryHistory` and extracts `recovery.messages`, the durable transcript of all prior interactions.

### The `planResume` Classification Engine

The core recovery logic resides in `planResume` (lines 94–126 of [`resume.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/resume.ts)). It sanitizes the transcript tail and classifies recovery actions:

| Tail State | `ResumeAction.kind` | Behavior |
|:---|:---|:---|
| Empty message array | `'start'` | Fresh session initiation with original prompt |
| Ends with `user` message or `toolResult` containing successful `ask_user`/`create_skill` | `'already-complete'` | Session paused awaiting external input; do not auto-respond |
| Ends with `assistant` message having **pending tool calls** | `'continue'` + `repairedToolCalls` | Resume mid-execution, re-issuing dangling calls |
| Ends with completed `assistant` turn (no pending calls) | `'already-complete'` | Terminal state; finalize as successful |

The algorithm **strips incomplete assistant frames**—messages that are empty, aborted, or length-truncated—preventing the LLM from receiving malformed context.

---

## Re-Issuing Dangling Tool Calls

For `'continue'` cases, the runner invokes **`repairOrphanedToolCalls`** from [`lib/server/agent-runtime/tool-call-integrity.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/server/agent-runtime/tool-call-integrity.ts). This utility injects **synthetic "interrupted" tool results** for each dangling call ID identified by `planResume`:

```typescript
const contextRepair = repairOrphanedToolCalls(plannedMessages);
const repairedToolCallIds = [
  ...new Set([
    ...(plan.kind === 'continue' ? plan.repairedToolCalls : []),
    ...contextRepair.repairedToolCalls,
  ]),
];

```

The Pi SDK (`agent.continue()`) perceives these as **failed tool executions**, prompting automatic retry. Tool implementations **must be idempotent** to safely handle this at-least-once execution guarantee.

---

## Finalization and Re-Queueing Logic

### Session Completion

Terminal events trigger `store.finishSession`. If lease loss occurred during the run, the commit aborts silently; the next worker will repeat the full resume flow. This ensures **exactly-once finalization semantics** modulo lease ownership.

### Undelivered Message Handling

When sessions terminate without delivering all pending user messages, `requeueIfUndelivered` (via `planUndeliveredRequeue`) evaluates whether to **reset the attempt counter** for backoff purposes or **immediate retry** based on divergence between logged messages and last confirmed delivery sequence.

---

## Practical Examples

### Simulating Lease Loss

Test lease detection without actual crashes:

```typescript
// Active runner holds sessionMeta with valid lease
const runnerHandle = await startRunner(sessionMeta);

// Force external lease acquisition
await store.forceLeaseSteal(sessionMeta.id, 'other-worker-id');

// Heartbeat detects loss within two intervals
await new Promise((resolve) => setTimeout(resolve, config.heartbeatIntervalMs * 2));
await runnerHandle.stop(); // Graceful cleanup

```

### Triggering Crash Recovery Manually

```typescript
// 1. Acquire session ownership
const meta = await claimSession('my-session-id');

// 2. Persist partial tool execution without completion
await store.appendRunEvent(meta.id, meta.workerId, {
  ts: Date.now(),
  attempt: meta.attempt,
  type: 'tool_execution_start',
  data: { toolName: 'generate_image', toolCallId: 't-42' },
});
// Deliberately omit tool_execution_end/toolResult

// 3. Release lease to simulate worker death
await store.releaseLease(meta.id, meta.workerId);

// 4. New worker automatically resumes and re-issues generate_image
await runSession(newRunContext, await claimSession(meta.id));

```

### Detecting `ask_user` Pause States

```typescript
const transcript = [
  { role: 'user', content: 'What do you think?' },
  { role: 'assistant', content: [{ type: 'toolCall', id: 'tc-1', name: 'ask_user' }] },
  { role: 'toolResult', toolName: 'ask_user', toolCallId: 'tc-1', isError: false },
];

const action = planResume(transcript);
console.log(action.kind); // 'already-complete' — awaiting human response

```

---

## Key Implementation Files

| File | Responsibility |
|:---|:---|
| [`lib/server/agent-runtime/runner.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/server/agent-runtime/runner.ts) | Core state machine: lease claims, heartbeat coordination, event loop, lease-loss detection, LLM interaction |
| [`lib/server/agent-runtime/resume.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/server/agent-runtime/resume.ts) | Post-crash transcript analysis, tail classification, `planResume` implementation |
| [`lib/server/agent-runtime/store.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/server/agent-runtime/store.ts) | PostgreSQL abstraction: `claimSession`, `heartbeat`, `releaseLease`, append operations |
| [`lib/server/agent-runtime/tool-call-integrity.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/server/agent-runtime/tool-call-integrity.ts) | Synthetic result injection for orphaned tool calls |
| [`lib/server/agent-runtime/event-notify-bus.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/server/agent-runtime/event-notify-bus.ts) | NOTIFY-based subscription for external event wakeups during lease hold |

---

## Summary

- **Lease coordination** combines PostgreSQL-backed claims with 5-second heartbeat refreshes and write-time validation via `leaseMatches`.

- **Crash recovery** rebuilds runnable state from durable history using `planResume`, which classifies session tails into start, continue, or already-complete actions.

- **Tool-call integrity** is preserved through synthetic interruption results, enabling idempotent retry of dangling operations.

- **Concurrent safety** is guaranteed by immediate abort on any lease-loss detection, whether from heartbeat failure or rejected durable writes.

---

## Frequently Asked Questions

### How does the runner prevent two workers from processing the same session simultaneously?

The runner acquires an exclusive lease through `claimSession` in [`lib/server/agent-runtime/store.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/server/agent-runtime/store.ts), validates ownership via `leaseMatches` on every critical operation, and maintains continuous heartbeat renewal. Any competing claim revokes the prior lease, causing heartbeat failures that trigger immediate local abort.

### What happens if a worker crashes mid-tool-execution?

The next acquiring worker loads the persisted transcript through `loadEntryHistory`, calls `planResume` to identify dangling tool calls, and uses `repairOrphanedToolCalls` to inject synthetic failure results. The LLM then automatically retries these calls with full context intact.

### Why must tools be idempotent in this system?

Because crash recovery may re-issue tool calls that partially executed before the original worker died. The `repairOrphanedToolCalls` mechanism treats all dangling calls as interrupted, causing automatic retry. Idempotent implementations ensure correct behavior despite multiple execution attempts.

### How does the runner distinguish between a fresh session and a resumed one after crash?

`planResume` analyzes the transcript tail: an empty history yields `'start'`; presence of `ask_user` or `create_skill` success results yields `'already-complete'`; incomplete assistant turns with pending tool calls yield `'continue'`. This classification drives subsequent runner behavior without external state.