# How OpenMAIC Handles Crash Recovery for Agent Sessions

> Discover how OpenMAIC ensures agent session continuity with robust crash recovery. Learn about its three-stage pipeline: detection, event emission, and client-side repair.

- Repository: [MAIC/OpenMAIC](https://github.com/THU-MAIC/OpenMAIC)
- Tags: how-to-guide
- Published: 2026-09-05

---

**OpenMAIC handles crash recovery for agent sessions through a three-stage pipeline: detecting and classifying the crash tail via `planResume()`, emitting a `session_resumed` event with repair metadata, and reconciling orphaned tool calls on the client to maintain UI consistency.**

Crash recovery for agent sessions ensures that users never lose progress when a worker process dies unexpectedly. In the OpenMAIC repository, this capability is implemented through a coordinated server-client protocol that inspects durable transcripts, classifies the interruption type, and repairs the UI state without requiring manual intervention.

## The Three-Stage Crash Recovery Pipeline

OpenMAIC's recovery workflow coordinates three distinct components across the runtime and frontend to restore session continuity.

### 1. Detect and Classify the Crash Tail

When a worker dies, the persistent transcript may end with an incomplete assistant turn or a dangling tool call. The `planResume()` function in [`lib/server/agent-runtime/resume.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/server/agent-runtime/resume.ts) inspects the last durable messages, strips any empty or interrupted assistant frames, and classifies the tail into one of three categories:

- **'start'** – No usable history exists; begin a fresh run
- **'continue'** – Resume from the last durable point, repairing orphaned tool calls
- **'already-complete'** – The run was already terminal (e.g., a successful `ask_user` result)

This classification determines whether the session requires resurrection or has already reached a terminal state.

### 2. Emit the Session Resumed Event

The runner in [`lib/server/agent-runtime/runner.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/server/agent-runtime/runner.ts) reads the `ResumeAction` returned by `planResume()`. If the session is not already complete, it emits a `session_resumed` lifecycle event that carries critical recovery metadata:

```typescript
emit(LIFECYCLE.sessionResumed, {
  workerId: WORKER_ID,
  pid: process.pid,
  attempt,
  reason: resumeAction.kind === 'start' ? 'follow_up' : 'crash',
  transcriptMessages: modelMessages.length,
  repairedToolCalls: resumeAction.kind === 'continue' ? resumeAction.repairedToolCalls : [],
});

```

This payload includes the `reason` field (distinguishing between `crash` and `follow_up`), the count of durable transcript messages, and an array of `repairedToolCalls` containing IDs of tool calls orphaned by the crash.

### 3. Repair Orphaned Tool Cards on the Client

On the frontend, the workbench fold receives the `session_resumed` event in [`lib/workbench/session-store.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/workbench/session-store.ts). It processes `repairedToolCalls` to mark any still-running tool cards as **failed**, inserting an explicit error message such as "worker interrupted":

```typescript
if (event.type === 'session_resumed' && event.data.repairedToolCalls) {
  const repaired = new Set(event.data.repairedToolCalls);
  chat = settleRunningToolCards(chat, {
    ts: event.ts,
    reason: defaultWorkbenchTranslator('workbench.system.workerInterrupted'),
    onlyCallIds: repaired,
    onGeneratingCardSettled: () => { next.generatingOrder = null; },
  });
}

```

If the orphaned call was a page-generating tool, the generating marker is cleared so the UI knows the page was not produced. This prevents "running" spinners from persisting indefinitely.

## Core Design Principles

The crash recovery mechanism relies on four architectural guarantees that ensure consistency across interruptions.

**Crash-Consistent Transcript** – The durable transcript is written after each turn or tool boundary. Even if the worker dies mid-tool, the transcript up to the last complete frame remains intact.

**Idempotent Tool Calls** – Tools are required to be idempotent (e.g., `putScene` uses deterministic IDs). When a tool call is repaired, the model re-issues the call, guaranteeing at-least-once execution without duplication.

**Deterministic Resume Point** – `planResume()` removes ambiguous frames (empty assistant messages or aborted generations) so the resumed run starts from a legally continuable state.

**UI Consistency** – By settling running tool cards on the client side, the system ensures UI elements reflect the true outcome of interrupted operations rather than stale "in-progress" states.

## Implementation Examples

### Server-Side Resume Planning

To determine how to resume a session after detecting a failure:

```typescript
import { planResume } from '@/lib/server/agent-runtime/resume';

const transcript = await store.getTranscript(sessionId);
const resumeAction = planResume(transcript);

// resumeAction.kind can be:
// - 'start'            → start a fresh run
// - 'continue'         → resume with repairedToolCalls
// - 'already-complete' → nothing to do, the run is finished

```

### Runner Event Emission Logic

The runner branches based on the resume classification:

```typescript
if (resumeAction.kind === 'start') {
  emit(LIFECYCLE.sessionStart, { /* … */ });
} else {
  emit(LIFECYCLE.sessionResumed, {
    workerId: WORKER_ID,
    pid: process.pid,
    attempt,
    reason: resumeAction.kind === 'start' ? 'follow_up' : 'crash',
    transcriptMessages: modelMessages.length,
    repairedToolCalls: resumeAction.kind === 'continue' ? resumeAction.repairedToolCalls : [],
  });
}

```

## Summary

- **Crash detection** occurs in [`lib/server/agent-runtime/resume.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/server/agent-runtime/resume.ts) via `planResume()`, which classifies the transcript tail as 'start', 'continue', or 'already-complete'
- **Event propagation** happens through [`lib/server/agent-runtime/runner.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/server/agent-runtime/runner.ts), which emits `session_resumed` with `repairedToolCalls` metadata identifying orphaned operations
- **UI reconciliation** is handled in [`lib/workbench/session-store.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/workbench/session-store.ts) using `settleRunningToolCards()` to mark interrupted tools as failed and clear generating markers
- **Idempotency requirements** ensure that repaired tool calls can be safely re-executed without side effects
- **Durable transcripts** guarantee that no progress is lost even when workers terminate mid-operation

## Frequently Asked Questions

### What happens to running tool calls when a worker crashes?

Running tool calls are identified as orphaned operations when `planResume()` classifies the crash tail. Their IDs are included in the `repairedToolCalls` array within the `session_resumed` event. The frontend then marks these specific tool cards as failed with an error message, preventing indefinite "running" spinners.

### How does OpenMAIC ensure no duplicate tool executions after recovery?

OpenMAIC requires all tools to be **idempotent** by design. When the session resumes, the model re-issues the interrupted tool calls using deterministic identifiers. Because the operations are idempotent (such as `putScene` with deterministic IDs), re-execution produces the same result without creating duplicates.

### Where is the crash recovery logic implemented in the codebase?

The logic spans three primary files: [`lib/server/agent-runtime/resume.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/server/agent-runtime/resume.ts) contains the classification algorithm, [`lib/server/agent-runtime/runner.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/server/agent-runtime/runner.ts) orchestrates event emission, and [`lib/workbench/session-store.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/workbench/session-store.ts) handles frontend state reconciliation.

### What are the possible resume states after a crash?

`planResume()` returns three possible states: **'start'** for sessions with no usable history, **'continue'** for sessions that can resume from the last durable point with repaired tool calls, and **'already-complete'** for sessions that had already reached a terminal state (such as a completed `ask_user` result) before the crash occurred.