Prime Agent Worker Recovery Strategy: How the Supervisor Restores Crashed Sessions

Prime Agent recovers crashed workers by replaying an append-only JSON-L journal that logs every LLM request, response, and acknowledgment, allowing the supervisor to reconstruct session state and resume without duplicating tool calls.

Prime Agent is an open-source coding agent that isolates each user session in a dedicated worker process. When that worker exits unexpectedly—whether from a crash, out-of-memory kill, or forced termination—the supervisor daemon executes a deterministic recovery protocol. This article explains the complete recovery strategy implemented in the PrimeIntellect-ai/prime-agent repository, from journal creation to state reconstruction.

---

Core Components of the Recovery System

The recovery strategy depends on two cooperating parts: a per-worker journal that persists interaction history, and a supervisor routine that interprets that journal to restore execution context.

WorkerRecoveryJournal: The Append-Only Event Log

The WorkerRecoveryJournal class implements a line-delimited JSON (JSON-L) log that records every meaningful event in a worker's lifecycle. Located in packages/coding-agent/src/modes/daemon/worker-recovery-journal.ts, this component guarantees that the supervisor can always reconstruct the worker's state up to the point of failure.

Each journal file follows a strict naming convention. By default, the worker creates a temporary directory with the prefix prime-agent-worker-recovery- and writes to worker.recovery.jsonl inside it. You can override this path by setting the environment variable PRIME_AGENT_INTERNAL_OWNED_RECOVERY_DESCRIPTOR before starting the worker.

// Worker initialization reads the environment variable
process.env.PRIME_AGENT_INTERNAL_OWNED_RECOVERY_DESCRIPTOR =
  "/custom/path/session-42.recovery.jsonl";

The journal accepts three event types, each written as a single JSON line:

• request — An outgoing LLM call with a stable UUID, model name, and prompt parameters
• response — The corresponding LLM output, linked by matching UUID
• ack — Confirmation that a tool call or side effect was received and processed

This structure makes the journal idempotent: replaying the same sequence multiple times produces identical state, because each event carries its own identity and the supervisor can detect duplicates.

Supervisor Recovery Routine

The supervisor, implemented in packages/coding-agent/src/modes/daemon/supervisor.ts, monitors worker health through PID tracking and heartbeat timeouts. When it detects a failure, it invokes recoverWorker() to orchestrate restoration.

The recovery flow proceeds through these validated stages:

1. Locate the journal — The supervisor reads the worker's configured recovery descriptor path from its internal state
2. Validate format version — The parser rejects journals with recovery format version: 2 or higher, surfacing a clear error for unsupported schemas
3. Rebuild state — The supervisor reconstructs the worker's internal queues, pending tool calls, heartbeat timers, and session metadata
4. Spawn replacement — A new worker process receives the reconstructed state via injection
5. Resume execution — The new worker replays journal entries to reach the pre-crash position, then continues with fresh LLM interactions
6. Cleanup — Upon successful acknowledgment, the supervisor removes the temporary journal file

The implementation handles edge cases including concurrent recovery attempts and temporary ownership changes. If a second recovery starts while another is in progress, the system either defers or cancels the redundant attempt according to the recoveryEligibilityInvalidations logic.

---

State Reconstruction Details

The supervisor's state reconstruction targets three critical data structures:

Structure	Purpose	Source in Journal
Queued actions	FIFO-ordered pending LLM calls and tool executions	Sequential request entries without matching ack
Last complete snapshot	Checkpoint for "last-known-good" recovery	Periodic metadata entries or final consistent state
Deferred recoveries	Recovery attempts that must survive ownership transfers	Ownership change markers with pending flags

The reconstruction process preserves stable identifiers for all in-flight operations. This prevents duplicate tool calls when the journal is replayed, because the new worker recognizes that a given UUID has already been processed.

---

Code Example: Journal Entry Format

Below is a representative snippet from a worker recovery journal:

{"type":"request","id":"c4b9a1e2-7f1d-4d5b-9a3e-12f34ab56789","model":"gpt-4o","prompt":"Implement a binary search tree in Rust"}
{"type":"response","id":"c4b9a1e2-7f1d-4d5b-9a3e-12f34ab56789","content":"Here's a Rust implementation...","finish_reason":"stop"}
{"type":"ack","id":"c4b9a1e2-7f1d-4d5b-9a3e-12f34ab56789","ack_type":"tool_call","tool_name":"write_file"}

The supervisor parses these lines sequentially, building an in-memory representation of the worker's queues and pending operations.

---

Validation Through Test Coverage

The recovery system is extensively validated across multiple test files in packages/coding-agent/test/:

• worker-recovery-journal.test.ts — Unit tests for journal integrity, format validation, and idempotent replay semantics
• daemon-supervisor-process.test.ts — End-to-end verification of worker crash detection and full recovery cycles
• daemon-supervisor-heartbeats.test.ts — Confirmation that the last complete worker snapshot is correctly selected during heartbeat-timeout recovery
• daemon-supervisor-heartbeats.test.ts (deferred recovery cases) — Verification that deferred recoveries survive ownership loss and are properly cleaned up after reacquisition

These test suites enforce that recovery remains low-latency (no full session restart) and safe (no duplication of observable side effects).

---

Configuration and Environment Variables

Variable	Effect	Default
PRIME_AGENT_INTERNAL_OWNED_RECOVERY_DESCRIPTOR	Overrides the journal file path	Auto-generated temp directory
PRIME_AGENT_INTERNAL_WORKER_RECOVERY_DESCRIPTOR	Legacy reference for descriptor format	Version 1 schema

Setting the owned descriptor variable allows operators to place recovery journals on durable storage, enabling recovery even when the default temp directory is cleared between restarts.

---

Summary

• Prime Agent worker recovery uses an append-only JSON-L journal to persist every LLM interaction
• The WorkerRecoveryJournal class in worker-recovery-journal.ts manages event logging and format validation
• The recoverWorker function in supervisor.ts drives crash detection, state reconstruction, and worker replacement
• Recovery is idempotent and handles concurrent attempts through eligibility invalidation logic
• Comprehensive tests validate correctness across crash, OOM, and heartbeat failure scenarios

---

Frequently Asked Questions

What triggers the Prime Agent worker recovery process?

Recovery activates when the supervisor detects that a worker's process ID no longer exists, or when a heartbeat timeout occurs. The supervisor continuously polls worker health; any ungraceful exit triggers the recoverWorker() routine.

Can I relocate the recovery journal to persistent storage?

Yes. Set the environment variable PRIME_AGENT_INTERNAL_OWNED_RECOVERY_DESCRIPTOR to a durable path before starting the worker. The journal will be written to that location, allowing recovery even if the default temporary directory is lost.

What happens if the recovery journal is corrupted?

The supervisor validates the journal format before attempting reconstruction. If it encounters an unsupported version or unparseable entry, it aborts recovery and surfaces an error to the user. The session terminates rather than risk undefined behavior from partial state.

Does replaying the journal duplicate tool calls?

No. The journal uses stable UUIDs for every request and acknowledgment. During reconstruction, the supervisor tracks which operations have been acknowledged. Only unacknowledged requests are reissued to the new worker, preventing duplicate side effects.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}