How the Worker Recovery Journal Handles Process Failures in Prime Agent

The Prime Agent Worker Recovery Journal is an append-only, crash-safe persistence layer that records worker activity to a JSON Lines file, enabling automatic recovery of active sessions after process crashes or supervisor-initiated restarts.

The worker recovery journal is a critical resilience mechanism in the Prime Agent daemon. When long-running coding sessions are active, unexpected failures must not result in lost work or inconsistent state. This article examines how PrimeIntellect-ai/prime-agent implements reliable failure recovery through lightweight on-disk journaling.

What the Worker Recovery Journal Stores

Each journal entry is a compact JSON object tracking a specific worker state transition. According to the source code in worker-recovery-journal.ts, records contain six key fields:

• activeSessionId — The daemon-wide identifier for the session being acted upon
• sessionId — The underlying session file's UUID
• sessionFile (optional) — Absolute path to the persisted session on disk
• busy — Boolean flag indicating whether a long-running operation is in progress
• operation — Human-readable event description (e.g., "prompt_accepted", "session_shutdown")
• recordedAt — ISO-8601 timestamp of the write

The journal captures meaningful lifecycle events: session creation, prompt acceptance, follow-up queuing, and graceful shutdown.

Atomic Write Mechanism for Crash Safety

The journal guarantees durability through synchronous, append-only writes. The implementation in worker-recovery-journal.ts follows this sequence:

// Simplified from worker-recovery-journal.ts
const fd = openSync(this.filePath, "a");
writeSync(fd, JSON.stringify(record) + "\n");
fsyncSync(fd);
closeSync(fd);
chmodSync(this.filePath, 0o600);

This pattern ensures that even if the worker process terminates mid-write, previously committed entries remain intact. The fsyncSync call forces the operating system to flush buffers to physical storage, while 0o600 permissions restrict read access to the file owner.

Automatic Compaction to Control File Growth

The journal implements intelligent cleanup to prevent unbounded growth. After every write, the system checks worker-recovery-journal.ts for the compaction condition:

Compaction triggers when all tracked sessions are idle (busy === false). When this occurs, the entire file is rewritten to retain only the latest record per activeSessionId.

This design maintains O(n) read performance on recovery while ensuring the on-disk footprint remains proportional to active session count rather than total historical operations.

Recovery Flow: From Crash to Continuation

The worker recovery journal integrates tightly with the daemon's supervisor architecture. The recovery process follows three phases:

1. Journal Initialization

When DAEMON_WORKER_RECOVERY_JOURNAL_ENV is set, the daemon instantiates the journal in daemon-mode.ts:

const recoveryJournalPath = process.env[DAEMON_WORKER_RECOVERY_JOURNAL_ENV];
if (options.worker && recoveryJournalPath) {
  this.recoveryJournal = new WorkerRecoveryJournal(recoveryJournalPath);
}

2. Continuous State Recording

Throughout the worker lifecycle, daemon-mode.ts calls recordWorkerRecoveryState at critical transition points:

private recordWorkerRecoveryState(
  state: ActiveSessionState,
  operation: string,
  busyOverride?: boolean,
): void {
  if (!this.recoveryJournal) return;
  const session = state.runtime.session;
  const busy =
    busyOverride ?? (isActiveSessionBusy(state) || session.isRetrying || session.hasAcceptedPromptInFlight);
  
  this.recoveryJournal.record({
    activeSessionId: state.activeSessionId,
    sessionId: session.sessionId,
    ...(session.sessionFile ? { sessionFile: session.sessionFile } : {}),
    busy,
    operation,
  });
}

3. Post-Crash Recovery Invocation

When the supervisor detects a worker failure via launchReplacementSupervisor, the replacement process rehydrates state through the static readLatest method:

import { WorkerRecoveryJournal } from "./worker-recovery-journal.js";

const path = process.env[DAEMON_WORKER_RECOVERY_JOURNAL_ENV]!;
const latestRecords = WorkerRecoveryJournal.readLatest(path);

for (const rec of latestRecords) {
  if (rec.busy) {
    // Re-attach to session or notify supervisor of in-flight work
    supervisor.rehydrateSession(rec.activeSessionId, rec.sessionFile);
  }
}

Supervisor Integration for End-to-End Reliability

The worker recovery journal does not operate in isolation. The Prime Agent supervisor (daemon-supervisor-monitor.test.ts) monitors worker health through process signals and heartbeat timeouts. Upon detecting a crash:

1. The supervisor captures the exit code and signal
2. A replacement worker is spawned with the same DAEMON_WORKER_RECOVERY_JOURNAL_ENV value
3. The new worker parses the journal to identify which sessions were busy at failure time
4. Client connections are re-attached to rehydrated sessions without data loss

This architecture decouples failure detection from recovery logic, allowing the journal format to evolve independently of supervisor behavior.

Key Implementation Files

File	Responsibility
packages/coding-agent/src/modes/daemon/worker-recovery-journal.ts	Core journal: parsing, atomic append, compaction, and readLatest API
packages/coding-agent/src/modes/daemon/daemon-mode.ts	Integration point: instantiates journal and calls recordWorkerRecoveryState
packages/coding-agent/test/worker-recovery-journal.test.ts	Unit tests for parsing, appending, and compaction edge cases
packages/coding-agent/test/daemon-supervisor-monitor.test.ts	Integration tests for supervisor-driven recovery scenarios

Summary

• The worker recovery journal provides append-only, crash-safe persistence for Prime Agent worker state
• Atomic writes with fsyncSync guarantee durability even during catastrophic failures
• Automatic compaction keeps file size bounded while preserving latest state per session
• Supervisor integration enables seamless replacement of crashed workers without client-visible interruption
• Static readLatest method allows recovery code to reconstruct in-flight work without instantiating a full journal object

Frequently Asked Questions

What happens if the journal file is corrupted?

Partial writes are truncated during parseRecords in worker-recovery-journal.ts. The parser reads JSON lines sequentially and stops at the first parse failure, treating all prior valid entries as authoritative. This means at most one record is lost—the one being written during the crash.

Can multiple workers write to the same journal simultaneously?

No. The journal is designed for single-writer semantics. Each worker process receives a unique journal path via DAEMON_WORKER_RECOVERY_JOURNAL_ENV, ensuring isolation. The supervisor coordinates handoff during replacement to prevent concurrent access.

How does the journal affect performance during normal operation?

Writes are synchronous but batched by naturally low-frequency events (session creation, prompt acceptance). The fsyncSync overhead is typically sub-millisecond for SSD-backed storage. Compaction occurs only when all sessions idle, so busy workers never trigger rewrite costs.

What distinguishes the worker recovery journal from general logging?

The journal is structured for machine recovery, not human debugging. It maintains latest-state semantics through compaction, uses minimal fields for deterministic reconstruction, and exposes a purpose-built readLatest API. General application logs remain separate and unbounded for observability purposes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}