How Prime Agent's Snapshot and Recovery Mechanism Handles Worker Crashes

Prime Agent uses an append-only journal to track session state and a chunked snapshot cache to persist transcript data, allowing the supervisor to restart workers and resume sessions without losing context after crashes.

Prime Agent's daemon architecture isolates long-running worker processes from the main supervisor to ensure resilience against failures. When a worker crashes unexpectedly, the system reconstructs the lost session state from two complementary persistence layers. This article examines the implementation details of Prime Agent's snapshot and recovery mechanism based on the source code in the PrimeIntellect-ai/prime-agent repository.

The Two-Layer Persistence Architecture

Prime Agent maintains durability through two distinct storage mechanisms that work together during recovery:

• WorkerRecoveryJournal: A line-oriented JSON log that records the latest status of every active session, including the busy flag, current operation, and optional snapshot file path. When the supervisor starts, it calls WorkerRecoveryJournal.readLatest to discover any session marked busy at the time of the crash.

• SnapshotTranscriptCache: A chunked buffer that stores transcript data streamed from the worker. While the worker is alive, it sends snapshot frames (session_snapshot_begin, session_snapshot_chunk, session_snapshot_end) that the supervisor buffers in the cache. If the worker dies mid-stream, the cache retains partially received chunks so the supervisor can finish receiving data or replace them with a newer generation.

Recording Worker State with WorkerRecoveryJournal

The WorkerRecoveryJournal class in packages/coding-agent/src/modes/daemon/worker-recovery-journal.ts maintains an append-only log of worker state changes. The supervisor instantiates the journal with a file path, creating the directory with 0o700 permissions if necessary.

export class WorkerRecoveryJournal {
  private readonly latest: Map<string, WorkerRecoveryRecord>;

  constructor(private readonly path: string) {
    mkdirSync(dirname(path), { recursive: true, mode: 0o700 });
    this.latest = parseRecords(path);
  }

  /** Called by the worker for every meaningful state change */
  record(input: Omit<WorkerRecoveryRecord, "version" | "recordedAt">): void {
    const previous = this.latest.get(input.activeSessionId);
    if (
      previous?.busy === input.busy &&
      previous.operation === input.operation &&
      previous.sessionFile === input.sessionFile
    ) {
      return;                     // duplicate – ignore
    }
    const record: WorkerRecoveryRecord = {
      version: 1,
      ...input,
      recordedAt: new Date().toISOString(),
    };
    this.append(record);          // append to the on‑disk log
    this.latest.set(record.activeSessionId, record);
    if ([...this.latest.values()].every((e) => !e.busy)) this.compact();
  }
}

Append-Only Logging and Compaction

The journal implements an append-only write strategy where each state change generates a new JSON line with version: 1 and an ISO timestamp. Before appending, the record method checks the in-memory latest map to suppress duplicate entries when the state hasn't actually changed.

When every session in the system becomes idle (busy: false), the journal automatically compacts itself by rewriting the file to contain only the minimal set of newest records, preventing unbounded file growth.

Streaming Session Snapshots

While a session is active, the worker streams its state to the supervisor through the protocol defined in packages/coding-agent/src/modes/daemon/daemon-worker-protocol.ts. The supervisor handles three distinct frame types in packages/coding-agent/src/modes/daemon/daemon-mode.ts:

import { createSnapshotTranscriptChunks, SNAPSHOT_TARGET_CHUNK_BYTES } from "./snapshot-transcript-cache.js";
import { WorkerRecoveryJournal } from "./worker-recovery-journal.js";

// When the worker sends a snapshot frame:
if (frame.type === "session_snapshot_begin") {
  // Register the snapshot start in the journal
  this.recoveryJournal.record({
    activeSessionId,
    sessionId,
    busy: true,
    operation: "snapshot",
    sessionFile: undefined,
  });
}

if (frame.type === "session_snapshot_chunk") {
  // Store the chunk in the transcript cache
  transcriptCache.appendChunk(frame);
}

if (frame.type === "session_snapshot_end") {
  // Mark the snapshot as finished
  this.recoveryJournal.record({
    activeSessionId,
    sessionId,
    busy: false,
    operation: "snapshot",
    sessionFile: transcriptCache.filePath,
  });
}

Chunked Transcript Caching

The snapshot is broken into chunks sized approximately to SNAPSHOT_TARGET_CHUNK_BYTES. The SnapshotTranscriptCache buffers these chunks in memory and writes a compact transcript file once the final session_snapshot_end frame arrives. This chunked approach ensures that even if the worker terminates unexpectedly between chunks, the supervisor retains whatever data was already transmitted.

Detecting Crashes and Restoring Sessions

When the supervisor detects worker termination through socket closure or PID exit, it executes a three-step recovery protocol:

1. Load the latest journal: The supervisor calls WorkerRecoveryJournal.readLatest(path) to identify any session still marked busy: true.

2. Spawn a replacement worker: The supervisor creates a new worker process with the environment variable DAEMON_WORKER_RECOVERY_JOURNAL_ENV pointing to the same journal file path.

3. Restore the session: The supervisor passes restoreActiveSessionId to the new worker, which invokes daemon-mode.restoreSessionFromSnapshot to replay the transcript file persisted in the cache.

If the crashed worker never completed its snapshot, the supervisor can still finish the operation because the snapshotCache retains any chunks already received. The recovery-catch-up tests in packages/coding-agent/test/suite/regressions/4677-snapshot-catchup-replacement.test.ts verify that a newer snapshot generation can supersede an incomplete one, ensuring the final restored state always reflects the most recent complete snapshot.

Key Source Files and Implementation Details

File	Role
packages/coding-agent/src/modes/daemon/worker-recovery-journal.ts	Persistent log of each session’s busy/operation state with append-only semantics and automatic compaction.
packages/coding-agent/src/modes/daemon/daemon-mode.ts	Core daemon loop that receives snapshot frames, updates the journal, and coordinates the transcript cache.
packages/coding-agent/src/modes/daemon/snapshot-transcript-cache.ts	Buffers chunked snapshot data and writes the final transcript file to disk.
packages/coding-agent/src/modes/daemon/daemon-worker-protocol.ts	Defines the wire-format including session_snapshot_begin, session_snapshot_chunk, and session_snapshot_end.
packages/coding-agent/test/suite/regressions/4677-snapshot-catchup-replacement.test.ts	Validates that incomplete snapshots can be replaced by newer generations and that journal compaction works correctly when workers become idle.

Summary

• WorkerRecoveryJournal maintains an append-only JSON log of session states, using in-memory deduplication and automatic compaction when all sessions are idle.
• SnapshotTranscriptCache receives chunked snapshot data from workers, buffering it until the complete transcript can be written to disk.
• On worker crash, the supervisor reads the journal to find busy sessions, spawns a new worker with DAEMON_WORKER_RECOVERY_JOURNAL_ENV, and passes restoreActiveSessionId to resume the session.
• The chunked snapshot protocol ensures that partial data survives crashes, and newer snapshot generations can replace incomplete ones during recovery.

Frequently Asked Questions

What happens if a worker crashes in the middle of creating a snapshot?

The supervisor retains any chunks already received in the SnapshotTranscriptCache. When the replacement worker starts, the supervisor can either finish receiving the remaining chunks from the new worker or replace the incomplete snapshot with a newer generation. The recovery tests confirm that the system always restores from the most recent complete snapshot.

How does WorkerRecoveryJournal prevent duplicate entries?

Before appending a record to the log, the record method compares the new input against the previous state stored in the in-memory latest Map. If the busy flag, operation, and session file path are identical, the method returns early without writing to disk, preventing redundant log entries.

What is the role of the DAEMON_WORKER_RECOVERY_JOURNAL_ENV environment variable?

This environment variable contains the file path to the journal. When the supervisor spawns a replacement worker after a crash, it sets this variable so the new worker can access the same recovery journal and determine which session needs to be restored via restoreActiveSessionId.

How does the system handle multiple simultaneous sessions?

The journal stores records keyed by activeSessionId in a Map, allowing it to track state for multiple concurrent sessions. The compaction logic only triggers when every session in the system is idle (busy: false), ensuring that active sessions always have their state preserved in the log.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}