How OpenMAIC Handles Crash Recovery for Agent Sessions

OpenMAIC handles crash recovery for agent sessions through a three-stage pipeline: detecting and classifying the crash tail via planResume(), emitting a session_resumed event with repair metadata, and reconciling orphaned tool calls on the client to maintain UI consistency.

Crash recovery for agent sessions ensures that users never lose progress when a worker process dies unexpectedly. In the OpenMAIC repository, this capability is implemented through a coordinated server-client protocol that inspects durable transcripts, classifies the interruption type, and repairs the UI state without requiring manual intervention.

The Three-Stage Crash Recovery Pipeline

OpenMAIC's recovery workflow coordinates three distinct components across the runtime and frontend to restore session continuity.

1. Detect and Classify the Crash Tail

When a worker dies, the persistent transcript may end with an incomplete assistant turn or a dangling tool call. The planResume() function in lib/server/agent-runtime/resume.ts inspects the last durable messages, strips any empty or interrupted assistant frames, and classifies the tail into one of three categories:

• 'start' – No usable history exists; begin a fresh run
• 'continue' – Resume from the last durable point, repairing orphaned tool calls
• 'already-complete' – The run was already terminal (e.g., a successful ask_user result)

This classification determines whether the session requires resurrection or has already reached a terminal state.

2. Emit the Session Resumed Event

The runner in lib/server/agent-runtime/runner.ts reads the ResumeAction returned by planResume(). If the session is not already complete, it emits a session_resumed lifecycle event that carries critical recovery metadata:

emit(LIFECYCLE.sessionResumed, {
  workerId: WORKER_ID,
  pid: process.pid,
  attempt,
  reason: resumeAction.kind === 'start' ? 'follow_up' : 'crash',
  transcriptMessages: modelMessages.length,
  repairedToolCalls: resumeAction.kind === 'continue' ? resumeAction.repairedToolCalls : [],
});

This payload includes the reason field (distinguishing between crash and follow_up), the count of durable transcript messages, and an array of repairedToolCalls containing IDs of tool calls orphaned by the crash.

3. Repair Orphaned Tool Cards on the Client

On the frontend, the workbench fold receives the session_resumed event in lib/workbench/session-store.ts. It processes repairedToolCalls to mark any still-running tool cards as failed, inserting an explicit error message such as "worker interrupted":

if (event.type === 'session_resumed' && event.data.repairedToolCalls) {
  const repaired = new Set(event.data.repairedToolCalls);
  chat = settleRunningToolCards(chat, {
    ts: event.ts,
    reason: defaultWorkbenchTranslator('workbench.system.workerInterrupted'),
    onlyCallIds: repaired,
    onGeneratingCardSettled: () => { next.generatingOrder = null; },
  });
}

If the orphaned call was a page-generating tool, the generating marker is cleared so the UI knows the page was not produced. This prevents "running" spinners from persisting indefinitely.

Core Design Principles

The crash recovery mechanism relies on four architectural guarantees that ensure consistency across interruptions.

Crash-Consistent Transcript – The durable transcript is written after each turn or tool boundary. Even if the worker dies mid-tool, the transcript up to the last complete frame remains intact.

Idempotent Tool Calls – Tools are required to be idempotent (e.g., putScene uses deterministic IDs). When a tool call is repaired, the model re-issues the call, guaranteeing at-least-once execution without duplication.

Deterministic Resume Point – planResume() removes ambiguous frames (empty assistant messages or aborted generations) so the resumed run starts from a legally continuable state.

UI Consistency – By settling running tool cards on the client side, the system ensures UI elements reflect the true outcome of interrupted operations rather than stale "in-progress" states.

Implementation Examples

Server-Side Resume Planning

To determine how to resume a session after detecting a failure:

import { planResume } from '@/lib/server/agent-runtime/resume';

const transcript = await store.getTranscript(sessionId);
const resumeAction = planResume(transcript);

// resumeAction.kind can be:
// - 'start'            → start a fresh run
// - 'continue'         → resume with repairedToolCalls
// - 'already-complete' → nothing to do, the run is finished

Runner Event Emission Logic

The runner branches based on the resume classification:

if (resumeAction.kind === 'start') {
  emit(LIFECYCLE.sessionStart, { /* … */ });
} else {
  emit(LIFECYCLE.sessionResumed, {
    workerId: WORKER_ID,
    pid: process.pid,
    attempt,
    reason: resumeAction.kind === 'start' ? 'follow_up' : 'crash',
    transcriptMessages: modelMessages.length,
    repairedToolCalls: resumeAction.kind === 'continue' ? resumeAction.repairedToolCalls : [],
  });
}

Summary

• Crash detection occurs in lib/server/agent-runtime/resume.ts via planResume(), which classifies the transcript tail as 'start', 'continue', or 'already-complete'
• Event propagation happens through lib/server/agent-runtime/runner.ts, which emits session_resumed with repairedToolCalls metadata identifying orphaned operations
• UI reconciliation is handled in lib/workbench/session-store.ts using settleRunningToolCards() to mark interrupted tools as failed and clear generating markers
• Idempotency requirements ensure that repaired tool calls can be safely re-executed without side effects
• Durable transcripts guarantee that no progress is lost even when workers terminate mid-operation

Frequently Asked Questions

What happens to running tool calls when a worker crashes?

Running tool calls are identified as orphaned operations when planResume() classifies the crash tail. Their IDs are included in the repairedToolCalls array within the session_resumed event. The frontend then marks these specific tool cards as failed with an error message, preventing indefinite "running" spinners.

How does OpenMAIC ensure no duplicate tool executions after recovery?

OpenMAIC requires all tools to be idempotent by design. When the session resumes, the model re-issues the interrupted tool calls using deterministic identifiers. Because the operations are idempotent (such as putScene with deterministic IDs), re-execution produces the same result without creating duplicates.

Where is the crash recovery logic implemented in the codebase?

The logic spans three primary files: lib/server/agent-runtime/resume.ts contains the classification algorithm, lib/server/agent-runtime/runner.ts orchestrates event emission, and lib/workbench/session-store.ts handles frontend state reconciliation.

What are the possible resume states after a crash?

planResume() returns three possible states: 'start' for sessions with no usable history, 'continue' for sessions that can resume from the last durable point with repaired tool calls, and 'already-complete' for sessions that had already reached a terminal state (such as a completed ask_user result) before the crash occurred.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}