# How Prime Agent's Snapshot and Recovery Mechanism Handles Worker Crashes

> Discover how Prime Agent's snapshot and recovery mechanism ensures session continuity and prevents data loss during worker crashes using its append-only journal and chunked snapshot cache.

- Repository: [Prime Intellect/prime-agent](https://github.com/PrimeIntellect-ai/prime-agent)
- Tags: how-to-guide
- Published: 2026-08-10

---

**Prime Agent uses an append-only journal to track session state and a chunked snapshot cache to persist transcript data, allowing the supervisor to restart workers and resume sessions without losing context after crashes.**

Prime Agent's daemon architecture isolates long-running worker processes from the main supervisor to ensure resilience against failures. When a worker crashes unexpectedly, the system reconstructs the lost session state from two complementary persistence layers. This article examines the implementation details of Prime Agent's snapshot and recovery mechanism based on the source code in the `PrimeIntellect-ai/prime-agent` repository.

## The Two-Layer Persistence Architecture

Prime Agent maintains durability through two distinct storage mechanisms that work together during recovery:

- **WorkerRecoveryJournal**: A line-oriented JSON log that records the latest status of every active session, including the busy flag, current operation, and optional snapshot file path. When the supervisor starts, it calls `WorkerRecoveryJournal.readLatest` to discover any session marked busy at the time of the crash.

- **SnapshotTranscriptCache**: A chunked buffer that stores transcript data streamed from the worker. While the worker is alive, it sends snapshot frames (`session_snapshot_begin`, `session_snapshot_chunk`, `session_snapshot_end`) that the supervisor buffers in the cache. If the worker dies mid-stream, the cache retains partially received chunks so the supervisor can finish receiving data or replace them with a newer generation.

## Recording Worker State with WorkerRecoveryJournal

The `WorkerRecoveryJournal` class in [`packages/coding-agent/src/modes/daemon/worker-recovery-journal.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/packages/coding-agent/src/modes/daemon/worker-recovery-journal.ts) maintains an append-only log of worker state changes. The supervisor instantiates the journal with a file path, creating the directory with `0o700` permissions if necessary.

```typescript
export class WorkerRecoveryJournal {
  private readonly latest: Map<string, WorkerRecoveryRecord>;

  constructor(private readonly path: string) {
    mkdirSync(dirname(path), { recursive: true, mode: 0o700 });
    this.latest = parseRecords(path);
  }

  /** Called by the worker for every meaningful state change */
  record(input: Omit<WorkerRecoveryRecord, "version" | "recordedAt">): void {
    const previous = this.latest.get(input.activeSessionId);
    if (
      previous?.busy === input.busy &&
      previous.operation === input.operation &&
      previous.sessionFile === input.sessionFile
    ) {
      return;                     // duplicate – ignore
    }
    const record: WorkerRecoveryRecord = {
      version: 1,
      ...input,
      recordedAt: new Date().toISOString(),
    };
    this.append(record);          // append to the on‑disk log
    this.latest.set(record.activeSessionId, record);
    if ([...this.latest.values()].every((e) => !e.busy)) this.compact();
  }
}

```

### Append-Only Logging and Compaction

The journal implements an append-only write strategy where each state change generates a new JSON line with `version: 1` and an ISO timestamp. Before appending, the `record` method checks the in-memory `latest` map to suppress duplicate entries when the state hasn't actually changed.

When every session in the system becomes idle (`busy: false`), the journal automatically compacts itself by rewriting the file to contain only the minimal set of newest records, preventing unbounded file growth.

## Streaming Session Snapshots

While a session is active, the worker streams its state to the supervisor through the protocol defined in [`packages/coding-agent/src/modes/daemon/daemon-worker-protocol.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/packages/coding-agent/src/modes/daemon/daemon-worker-protocol.ts). The supervisor handles three distinct frame types in [`packages/coding-agent/src/modes/daemon/daemon-mode.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/packages/coding-agent/src/modes/daemon/daemon-mode.ts):

```typescript
import { createSnapshotTranscriptChunks, SNAPSHOT_TARGET_CHUNK_BYTES } from "./snapshot-transcript-cache.js";
import { WorkerRecoveryJournal } from "./worker-recovery-journal.js";

// When the worker sends a snapshot frame:
if (frame.type === "session_snapshot_begin") {
  // Register the snapshot start in the journal
  this.recoveryJournal.record({
    activeSessionId,
    sessionId,
    busy: true,
    operation: "snapshot",
    sessionFile: undefined,
  });
}

if (frame.type === "session_snapshot_chunk") {
  // Store the chunk in the transcript cache
  transcriptCache.appendChunk(frame);
}

if (frame.type === "session_snapshot_end") {
  // Mark the snapshot as finished
  this.recoveryJournal.record({
    activeSessionId,
    sessionId,
    busy: false,
    operation: "snapshot",
    sessionFile: transcriptCache.filePath,
  });
}

```

### Chunked Transcript Caching

The snapshot is broken into chunks sized approximately to `SNAPSHOT_TARGET_CHUNK_BYTES`. The `SnapshotTranscriptCache` buffers these chunks in memory and writes a compact transcript file once the final `session_snapshot_end` frame arrives. This chunked approach ensures that even if the worker terminates unexpectedly between chunks, the supervisor retains whatever data was already transmitted.

## Detecting Crashes and Restoring Sessions

When the supervisor detects worker termination through socket closure or PID exit, it executes a three-step recovery protocol:

1. **Load the latest journal**: The supervisor calls `WorkerRecoveryJournal.readLatest(path)` to identify any session still marked `busy: true`.

2. **Spawn a replacement worker**: The supervisor creates a new worker process with the environment variable `DAEMON_WORKER_RECOVERY_JOURNAL_ENV` pointing to the same journal file path.

3. **Restore the session**: The supervisor passes `restoreActiveSessionId` to the new worker, which invokes `daemon-mode.restoreSessionFromSnapshot` to replay the transcript file persisted in the cache.

If the crashed worker never completed its snapshot, the supervisor can still finish the operation because the `snapshotCache` retains any chunks already received. The recovery-catch-up tests in [`packages/coding-agent/test/suite/regressions/4677-snapshot-catchup-replacement.test.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/packages/coding-agent/test/suite/regressions/4677-snapshot-catchup-replacement.test.ts) verify that a newer snapshot generation can supersede an incomplete one, ensuring the final restored state always reflects the most recent complete snapshot.

## Key Source Files and Implementation Details

| File | Role |
|------|------|
| [`packages/coding-agent/src/modes/daemon/worker-recovery-journal.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/packages/coding-agent/src/modes/daemon/worker-recovery-journal.ts) | Persistent log of each session’s busy/operation state with append-only semantics and automatic compaction. |
| [`packages/coding-agent/src/modes/daemon/daemon-mode.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/packages/coding-agent/src/modes/daemon/daemon-mode.ts) | Core daemon loop that receives snapshot frames, updates the journal, and coordinates the transcript cache. |
| [`packages/coding-agent/src/modes/daemon/snapshot-transcript-cache.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/packages/coding-agent/src/modes/daemon/snapshot-transcript-cache.ts) | Buffers chunked snapshot data and writes the final transcript file to disk. |
| [`packages/coding-agent/src/modes/daemon/daemon-worker-protocol.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/packages/coding-agent/src/modes/daemon/daemon-worker-protocol.ts) | Defines the wire-format including `session_snapshot_begin`, `session_snapshot_chunk`, and `session_snapshot_end`. |
| [`packages/coding-agent/test/suite/regressions/4677-snapshot-catchup-replacement.test.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/packages/coding-agent/test/suite/regressions/4677-snapshot-catchup-replacement.test.ts) | Validates that incomplete snapshots can be replaced by newer generations and that journal compaction works correctly when workers become idle. |

## Summary

- **WorkerRecoveryJournal** maintains an append-only JSON log of session states, using in-memory deduplication and automatic compaction when all sessions are idle.
- **SnapshotTranscriptCache** receives chunked snapshot data from workers, buffering it until the complete transcript can be written to disk.
- On worker crash, the supervisor reads the journal to find busy sessions, spawns a new worker with `DAEMON_WORKER_RECOVERY_JOURNAL_ENV`, and passes `restoreActiveSessionId` to resume the session.
- The chunked snapshot protocol ensures that partial data survives crashes, and newer snapshot generations can replace incomplete ones during recovery.

## Frequently Asked Questions

### What happens if a worker crashes in the middle of creating a snapshot?

The supervisor retains any chunks already received in the `SnapshotTranscriptCache`. When the replacement worker starts, the supervisor can either finish receiving the remaining chunks from the new worker or replace the incomplete snapshot with a newer generation. The recovery tests confirm that the system always restores from the most recent complete snapshot.

### How does WorkerRecoveryJournal prevent duplicate entries?

Before appending a record to the log, the `record` method compares the new input against the previous state stored in the in-memory `latest` Map. If the busy flag, operation, and session file path are identical, the method returns early without writing to disk, preventing redundant log entries.

### What is the role of the DAEMON_WORKER_RECOVERY_JOURNAL_ENV environment variable?

This environment variable contains the file path to the journal. When the supervisor spawns a replacement worker after a crash, it sets this variable so the new worker can access the same recovery journal and determine which session needs to be restored via `restoreActiveSessionId`.

### How does the system handle multiple simultaneous sessions?

The journal stores records keyed by `activeSessionId` in a Map, allowing it to track state for multiple concurrent sessions. The compaction logic only triggers when **every** session in the system is idle (`busy: false`), ensuring that active sessions always have their state preserved in the log.