How OpenMAIC Achieves Crash Recovery and Cancellation for Agent Sessions

OpenMAIC's workbench uses three complementary mechanisms—persistent event streams, periodic reconciliation, and stream-health monitoring—to guarantee that any agent session can be cancelled instantly and that the UI automatically recovers from crashes or network failures.

OpenMAIC, an open-source multi-agent intelligence framework from THU-MAIC, provides robust crash recovery and cancellation for agent sessions through a dedicated client architecture. The OwnerSessionClient class maintains a live, owner-level view of every session, ensuring that user-initiated cancellations propagate reliably and that temporary failures heal automatically without manual intervention.

The Three-Layer Resilience Architecture

The system combines push-based updates with defensive polling to create an eventually consistent, fault-tolerant session manager.

Event-Source Stream for Real-Time Updates

At the core is a persistent EventSource connection that pushes incremental session events. The client registers listeners for six event types in lib/workbench/owner-session-client.ts (lines 13-20):

• session_created
• session_status
• session_deleted
• session_active_stage
• session_cancel_requested
• session_title

This push channel delivers low-latency updates, including the critical cancellation signal that triggers immediate UI changes.

Periodic Full Reconciliation (60-Second Snapshots)

Every ~60 seconds (with jitter), the client executes a complete REST fetch of the session list via scheduleReconciliation (lines 41-49). This snapshot:

• Re-applies any events lost during stream degradation
• Serves as a safety net for malformed or missed events
• Guarantees eventual consistency even if the push stream fails permanently

Stream-Health Monitoring and Malformed-Event Guard

The client samples EventSource.readyState every 5 seconds. Two defensive triggers protect against inconsistent state:

Guard	Trigger	Action
Connection stall	9 consecutive CONNECTING samples (~45s)	Marks stream degraded via onStreamHealth(false) (lines 42-48)
Malformed events	5 unparsable payloads	Forces immediate full fetch (lines 33-39)

If an incoming event cannot be parsed as an OwnerSessionEvent, the client increments a counter and rebuilds state from the authoritative server snapshot.

Session Cancellation Flow

OpenMAIC implements a four-step cancellation pipeline that ensures the UI stays synchronized even when requests collide with terminal states.

Step 1: User Trigger

The front-end invokes the public helper cancelWorkbenchSession(sessionId) from lib/workbench/use-workbench-session.ts:

import { cancelWorkbenchSession } from '@/lib/workbench/use-workbench-session';

async function handleCancel(sessionId: string) {
  try {
    await cancelWorkbenchSession(sessionId);
    // UI updates automatically via OwnerSessionClient callbacks
  } catch (err) {
    console.error('Cancel failed', err);
  }
}

Step 2: API Request

The helper sends a POST /api/agent/{sessionId}/cancel request. The server updates the session status to cancelled and emits session_cancel_requested on the owner stream.

export async function cancelWorkbenchSession(sessionId: string): Promise<void> {
  const resp = await fetch(`/api/agent/${sessionId}/cancel`, { method: 'POST' });
  if (!resp.ok) {
    // Translate 409 (already cancelled) into silent success
    if (resp.status === 409) return;
    throw new Error(`Cancel request failed: ${resp.status}`);
  }
}

Step 3: Client Event Handling

OwnerSessionClient's onSessionEvent handler (lines 31-38) processes the cancellation like any other event: it updates the local cursor, pushes to the journal, and triggers a full fetch if needed:

private openStream(): void {
  // ...
  const onSessionEvent = (raw: Event) => {
    const event = parseData(raw);
    if (!isOwnerSessionEvent(event)) { /* malformed guard */ }
    this.cursor = event.id;
    if (event.type === 'session_cancel_requested') {
      this.journal.push(event);
      // UI receives updated status: 'cancelled'
    }
    // ...
  };
}

Step 4: UI Reflection

The onSessions callback delivers the updated session list. Components rendering the session (rail, classroom view) switch to the idle tone, as verified by presentWorkspaceSession('cancelled').tone in workspace-navigation.test.ts (line 30).

Collision Handling

If a cancel request races against an already-terminal status (e.g., succeeded), the client translates the server's 409 error into a cancelled terminal status. Test coverage in session-cancel-client.test.ts (lines 35-60) validates this conflict resolution.

Crash Recovery Scenarios

Failure Mode	Detection	Recovery
Stream loss/stall	Health monitor flags 45s CONNECTING state	UI shows degraded indicator; next reconciliation restores state
Missing/malformed events	5-parser-failure counter	Immediate full fetch rebuilds authoritative snapshot
Client crash/restart	N/A (fresh connection)	Initial reconciliation fetches complete session list
Server restart	Stream error/close	Automatic reconnection with exponential backoff; reconciliation fills gap

The 60-second periodic reconciliation serves as the ultimate fallback: even if the event stream never recovers, the client re-aligns with server state within one minute.

Key Source Files

File	Responsibility
lib/workbench/owner-session-client.ts	Core client: event handling, health monitoring, reconciliation scheduling (lines 13-52, 41-49, 42-48)
lib/workbench/use-workbench-session.ts	High-level helpers: cancelWorkbenchSession
pages/api/agent/[sessionId]/cancel.ts	Server endpoint: status update, event emission
tests/workbench/session-cancel-client.test.ts	Verifies terminal-status collision handling (lines 35-60)
tests/workbench/workspace-navigation.test.ts	Asserts cancelled session UI state (line 30)

Summary

• Three mechanisms—event streams, periodic reconciliation, and health monitors—provide defense in depth for crash recovery and cancellation for agent sessions in OpenMAIC.
• Cancellation propagates via session_cancel_requested events, with 409-race handling ensuring UI consistency.
• Stream degradation triggers visible indicators and automatic healing through full fetches.
• Malformed events and connection stalls have dedicated guards that prevent silent data loss.
• Periodic reconciliation guarantees eventual consistency regardless of streaming layer health.

Frequently Asked Questions

How does OpenMAIC handle cancellation when the session already finished?

The cancelWorkbenchSession helper checks for HTTP 409 responses from the server, which indicate the session is already in a terminal state. The client silently treats this as success and updates local state to cancelled, preventing UI inconsistency. This behavior is tested in session-cancel-client.test.ts (lines 35-60).

What happens if the EventSource connection drops for more than a minute?

The stream-health monitor (observeStreamHealth, lines 42-48) flags the connection as degraded after ~45 seconds of CONNECTING state. Meanwhile, the scheduled reconciliation (every ~60 seconds) performs a full REST fetch to rebuild session state. The UI may show a degraded indicator, but functionality recovers automatically without page reload.

Can users cancel sessions during network partitions?

Partially. The cancel request requires server confirmation to emit the session_cancel_requested event. If the network partition prevents the POST from reaching the server, the cancellation queues locally and retries. Once connectivity restores, either the cancel succeeds or the reconciliation fetch reveals the session's actual terminal status, resolving any ambiguity.

How does OpenMAIC prevent malformed events from corrupting session state?

The client parses every incoming payload as an OwnerSessionEvent. After five parsing failures, it forces an immediate full fetch via the reconciliation path (lines 33-39). This rebuilds state from the server's authoritative snapshot, discarding any corrupted incremental updates.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}