# How OpenMAIC Achieves Crash Recovery and Cancellation for Agent Sessions

> OpenMAIC ensures agent session crash recovery and cancellation using persistent event streams, periodic reconciliation, and stream-health monitoring. Learn how OpenMAIC guarantees reliable agent sessions.

- Repository: [MAIC/OpenMAIC](https://github.com/THU-MAIC/OpenMAIC)
- Tags: internals
- Published: 2026-09-06

---

**OpenMAIC's workbench uses three complementary mechanisms—persistent event streams, periodic reconciliation, and stream-health monitoring—to guarantee that any agent session can be cancelled instantly and that the UI automatically recovers from crashes or network failures.**

OpenMAIC, an open-source multi-agent intelligence framework from THU-MAIC, provides robust **crash recovery and cancellation for agent sessions** through a dedicated client architecture. The `OwnerSessionClient` class maintains a live, owner-level view of every session, ensuring that user-initiated cancellations propagate reliably and that temporary failures heal automatically without manual intervention.

## The Three-Layer Resilience Architecture

The system combines push-based updates with defensive polling to create an eventually consistent, fault-tolerant session manager.

### Event-Source Stream for Real-Time Updates

At the core is a persistent `EventSource` connection that pushes incremental session events. The client registers listeners for six event types in [`lib/workbench/owner-session-client.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/workbench/owner-session-client.ts) (lines 13-20):

- `session_created`
- `session_status`
- `session_deleted`
- `session_active_stage`
- **`session_cancel_requested`**
- `session_title`

This push channel delivers low-latency updates, including the critical cancellation signal that triggers immediate UI changes.

### Periodic Full Reconciliation (60-Second Snapshots)

Every ~60 seconds (with jitter), the client executes a complete REST fetch of the session list via `scheduleReconciliation` (lines 41-49). This snapshot:

- Re-applies any events lost during stream degradation
- Serves as a safety net for malformed or missed events
- Guarantees eventual consistency even if the push stream fails permanently

### Stream-Health Monitoring and Malformed-Event Guard

The client samples `EventSource.readyState` every 5 seconds. Two defensive triggers protect against inconsistent state:

| Guard | Trigger | Action |
|-------|---------|--------|
| **Connection stall** | 9 consecutive `CONNECTING` samples (~45s) | Marks stream degraded via `onStreamHealth(false)` (lines 42-48) |
| **Malformed events** | 5 unparsable payloads | Forces immediate full fetch (lines 33-39) |

If an incoming event cannot be parsed as an `OwnerSessionEvent`, the client increments a counter and rebuilds state from the authoritative server snapshot.

## Session Cancellation Flow

OpenMAIC implements a four-step cancellation pipeline that ensures the UI stays synchronized even when requests collide with terminal states.

### Step 1: User Trigger

The front-end invokes the public helper `cancelWorkbenchSession(sessionId)` from [`lib/workbench/use-workbench-session.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/workbench/use-workbench-session.ts):

```tsx
import { cancelWorkbenchSession } from '@/lib/workbench/use-workbench-session';

async function handleCancel(sessionId: string) {
  try {
    await cancelWorkbenchSession(sessionId);
    // UI updates automatically via OwnerSessionClient callbacks
  } catch (err) {
    console.error('Cancel failed', err);
  }
}

```

### Step 2: API Request

The helper sends a `POST /api/agent/{sessionId}/cancel` request. The server updates the session status to `cancelled` and emits `session_cancel_requested` on the owner stream.

```ts
export async function cancelWorkbenchSession(sessionId: string): Promise<void> {
  const resp = await fetch(`/api/agent/${sessionId}/cancel`, { method: 'POST' });
  if (!resp.ok) {
    // Translate 409 (already cancelled) into silent success
    if (resp.status === 409) return;
    throw new Error(`Cancel request failed: ${resp.status}`);
  }
}

```

### Step 3: Client Event Handling

`OwnerSessionClient`'s `onSessionEvent` handler (lines 31-38) processes the cancellation like any other event: it updates the local cursor, pushes to the journal, and triggers a full fetch if needed:

```ts
private openStream(): void {
  // ...
  const onSessionEvent = (raw: Event) => {
    const event = parseData(raw);
    if (!isOwnerSessionEvent(event)) { /* malformed guard */ }
    this.cursor = event.id;
    if (event.type === 'session_cancel_requested') {
      this.journal.push(event);
      // UI receives updated status: 'cancelled'
    }
    // ...
  };
}

```

### Step 4: UI Reflection

The `onSessions` callback delivers the updated session list. Components rendering the session (rail, classroom view) switch to the *idle* tone, as verified by `presentWorkspaceSession('cancelled').tone` in [`workspace-navigation.test.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/workspace-navigation.test.ts) (line 30).

### Collision Handling

If a cancel request races against an already-terminal status (e.g., `succeeded`), the client translates the server's 409 error into a `cancelled` terminal status. Test coverage in [`session-cancel-client.test.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/session-cancel-client.test.ts) (lines 35-60) validates this conflict resolution.

## Crash Recovery Scenarios

| Failure Mode | Detection | Recovery |
|------------|-----------|----------|
| **Stream loss/stall** | Health monitor flags 45s `CONNECTING` state | UI shows degraded indicator; next reconciliation restores state |
| **Missing/malformed events** | 5-parser-failure counter | Immediate full fetch rebuilds authoritative snapshot |
| **Client crash/restart** | N/A (fresh connection) | Initial reconciliation fetches complete session list |
| **Server restart** | Stream error/close | Automatic reconnection with exponential backoff; reconciliation fills gap |

The 60-second periodic reconciliation serves as the ultimate fallback: even if the event stream never recovers, the client re-aligns with server state within one minute.

## Key Source Files

| File | Responsibility |
|------|--------------|
| [`lib/workbench/owner-session-client.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/workbench/owner-session-client.ts) | Core client: event handling, health monitoring, reconciliation scheduling (lines 13-52, 41-49, 42-48) |
| [`lib/workbench/use-workbench-session.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/workbench/use-workbench-session.ts) | High-level helpers: `cancelWorkbenchSession` |
| `pages/api/agent/[sessionId]/cancel.ts` | Server endpoint: status update, event emission |
| [`tests/workbench/session-cancel-client.test.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/tests/workbench/session-cancel-client.test.ts) | Verifies terminal-status collision handling (lines 35-60) |
| [`tests/workbench/workspace-navigation.test.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/tests/workbench/workspace-navigation.test.ts) | Asserts cancelled session UI state (line 30) |

## Summary

- **Three mechanisms**—event streams, periodic reconciliation, and health monitors—provide defense in depth for **crash recovery and cancellation for agent sessions** in OpenMAIC.
- **Cancellation** propagates via `session_cancel_requested` events, with 409-race handling ensuring UI consistency.
- **Stream degradation** triggers visible indicators and automatic healing through full fetches.
- **Malformed events** and **connection stalls** have dedicated guards that prevent silent data loss.
- **Periodic reconciliation** guarantees eventual consistency regardless of streaming layer health.

## Frequently Asked Questions

### How does OpenMAIC handle cancellation when the session already finished?

The `cancelWorkbenchSession` helper checks for HTTP 409 responses from the server, which indicate the session is already in a terminal state. The client silently treats this as success and updates local state to `cancelled`, preventing UI inconsistency. This behavior is tested in [`session-cancel-client.test.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/session-cancel-client.test.ts) (lines 35-60).

### What happens if the EventSource connection drops for more than a minute?

The stream-health monitor (`observeStreamHealth`, lines 42-48) flags the connection as degraded after ~45 seconds of `CONNECTING` state. Meanwhile, the scheduled reconciliation (every ~60 seconds) performs a full REST fetch to rebuild session state. The UI may show a degraded indicator, but functionality recovers automatically without page reload.

### Can users cancel sessions during network partitions?

Partially. The cancel request requires server confirmation to emit the `session_cancel_requested` event. If the network partition prevents the POST from reaching the server, the cancellation queues locally and retries. Once connectivity restores, either the cancel succeeds or the reconciliation fetch reveals the session's actual terminal status, resolving any ambiguity.

### How does OpenMAIC prevent malformed events from corrupting session state?

The client parses every incoming payload as an `OwnerSessionEvent`. After five parsing failures, it forces an immediate full fetch via the reconciliation path (lines 33-39). This rebuilds state from the server's authoritative snapshot, discarding any corrupted incremental updates.