# Prime Agent Worker Recovery Strategy: How the Supervisor Restores Crashed Sessions

> Discover Prime Agent's recovery strategy for crashed workers. Learn how journal replay reconstructs session state and resumes operations without duplicate tool calls.

- Repository: [Prime Intellect/prime-agent](https://github.com/PrimeIntellect-ai/prime-agent)
- Tags: how-to-guide
- Published: 2026-08-15

---

**Prime Agent recovers crashed workers by replaying an append-only JSON-L journal that logs every LLM request, response, and acknowledgment, allowing the supervisor to reconstruct session state and resume without duplicating tool calls.**

Prime Agent is an open-source coding agent that isolates each user session in a dedicated worker process. When that worker exits unexpectedly—whether from a crash, out-of-memory kill, or forced termination—the supervisor daemon executes a deterministic recovery protocol. This article explains the complete recovery strategy implemented in the `PrimeIntellect-ai/prime-agent` repository, from journal creation to state reconstruction.

---

## Core Components of the Recovery System

The recovery strategy depends on two cooperating parts: a per-worker journal that persists interaction history, and a supervisor routine that interprets that journal to restore execution context.

### WorkerRecoveryJournal: The Append-Only Event Log

The `WorkerRecoveryJournal` class implements a line-delimited JSON (JSON-L) log that records every meaningful event in a worker's lifecycle. Located in [`packages/coding-agent/src/modes/daemon/worker-recovery-journal.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/packages/coding-agent/src/modes/daemon/worker-recovery-journal.ts), this component guarantees that the supervisor can always reconstruct the worker's state up to the point of failure.

Each journal file follows a strict naming convention. By default, the worker creates a temporary directory with the prefix `prime-agent-worker-recovery-` and writes to `worker.recovery.jsonl` inside it. You can override this path by setting the environment variable `PRIME_AGENT_INTERNAL_OWNED_RECOVERY_DESCRIPTOR` before starting the worker.

```typescript
// Worker initialization reads the environment variable
process.env.PRIME_AGENT_INTERNAL_OWNED_RECOVERY_DESCRIPTOR =
  "/custom/path/session-42.recovery.jsonl";

```

The journal accepts three event types, each written as a single JSON line:

- **`request`** — An outgoing LLM call with a stable UUID, model name, and prompt parameters
- **`response`** — The corresponding LLM output, linked by matching UUID
- **`ack`** — Confirmation that a tool call or side effect was received and processed

This structure makes the journal **idempotent**: replaying the same sequence multiple times produces identical state, because each event carries its own identity and the supervisor can detect duplicates.

### Supervisor Recovery Routine

The supervisor, implemented in [`packages/coding-agent/src/modes/daemon/supervisor.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/packages/coding-agent/src/modes/daemon/supervisor.ts), monitors worker health through PID tracking and heartbeat timeouts. When it detects a failure, it invokes `recoverWorker()` to orchestrate restoration.

The recovery flow proceeds through these validated stages:

1. **Locate the journal** — The supervisor reads the worker's configured recovery descriptor path from its internal state
2. **Validate format version** — The parser rejects journals with `recovery format version: 2` or higher, surfacing a clear error for unsupported schemas
3. **Rebuild state** — The supervisor reconstructs the worker's internal queues, pending tool calls, heartbeat timers, and session metadata
4. **Spawn replacement** — A new worker process receives the reconstructed state via injection
5. **Resume execution** — The new worker replays journal entries to reach the pre-crash position, then continues with fresh LLM interactions
6. **Cleanup** — Upon successful acknowledgment, the supervisor removes the temporary journal file

The implementation handles edge cases including concurrent recovery attempts and temporary ownership changes. If a second recovery starts while another is in progress, the system either defers or cancels the redundant attempt according to the `recoveryEligibilityInvalidations` logic.

---

## State Reconstruction Details

The supervisor's state reconstruction targets three critical data structures:

| Structure | Purpose | Source in Journal |
|-----------|---------|-------------------|
| **Queued actions** | FIFO-ordered pending LLM calls and tool executions | Sequential `request` entries without matching `ack` |
| **Last complete snapshot** | Checkpoint for "last-known-good" recovery | Periodic metadata entries or final consistent state |
| **Deferred recoveries** | Recovery attempts that must survive ownership transfers | Ownership change markers with pending flags |

The reconstruction process preserves **stable identifiers** for all in-flight operations. This prevents duplicate tool calls when the journal is replayed, because the new worker recognizes that a given UUID has already been processed.

---

## Code Example: Journal Entry Format

Below is a representative snippet from a worker recovery journal:

```json
{"type":"request","id":"c4b9a1e2-7f1d-4d5b-9a3e-12f34ab56789","model":"gpt-4o","prompt":"Implement a binary search tree in Rust"}
{"type":"response","id":"c4b9a1e2-7f1d-4d5b-9a3e-12f34ab56789","content":"Here's a Rust implementation...","finish_reason":"stop"}
{"type":"ack","id":"c4b9a1e2-7f1d-4d5b-9a3e-12f34ab56789","ack_type":"tool_call","tool_name":"write_file"}

```

The supervisor parses these lines sequentially, building an in-memory representation of the worker's queues and pending operations.

---

## Validation Through Test Coverage

The recovery system is extensively validated across multiple test files in `packages/coding-agent/test/`:

- [`worker-recovery-journal.test.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/worker-recovery-journal.test.ts) — Unit tests for journal integrity, format validation, and idempotent replay semantics
- [`daemon-supervisor-process.test.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/daemon-supervisor-process.test.ts) — End-to-end verification of worker crash detection and full recovery cycles
- [`daemon-supervisor-heartbeats.test.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/daemon-supervisor-heartbeats.test.ts) — Confirmation that the last complete worker snapshot is correctly selected during heartbeat-timeout recovery
- [`daemon-supervisor-heartbeats.test.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/daemon-supervisor-heartbeats.test.ts) (deferred recovery cases) — Verification that deferred recoveries survive ownership loss and are properly cleaned up after reacquisition

These test suites enforce that recovery remains **low-latency** (no full session restart) and **safe** (no duplication of observable side effects).

---

## Configuration and Environment Variables

| Variable | Effect | Default |
|----------|--------|---------|
| `PRIME_AGENT_INTERNAL_OWNED_RECOVERY_DESCRIPTOR` | Overrides the journal file path | Auto-generated temp directory |
| `PRIME_AGENT_INTERNAL_WORKER_RECOVERY_DESCRIPTOR` | Legacy reference for descriptor format | Version 1 schema |

Setting the owned descriptor variable allows operators to place recovery journals on durable storage, enabling recovery even when the default temp directory is cleared between restarts.

---

## Summary

- Prime Agent worker recovery uses an **append-only JSON-L journal** to persist every LLM interaction
- The **`WorkerRecoveryJournal`** class in [`worker-recovery-journal.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/worker-recovery-journal.ts) manages event logging and format validation
- The **`recoverWorker`** function in [`supervisor.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/supervisor.ts) drives crash detection, state reconstruction, and worker replacement
- Recovery is **idempotent** and handles **concurrent attempts** through eligibility invalidation logic
- **Comprehensive tests** validate correctness across crash, OOM, and heartbeat failure scenarios

---

## Frequently Asked Questions

### What triggers the Prime Agent worker recovery process?

Recovery activates when the supervisor detects that a worker's process ID no longer exists, or when a heartbeat timeout occurs. The supervisor continuously polls worker health; any ungraceful exit triggers the `recoverWorker()` routine.

### Can I relocate the recovery journal to persistent storage?

Yes. Set the environment variable `PRIME_AGENT_INTERNAL_OWNED_RECOVERY_DESCRIPTOR` to a durable path before starting the worker. The journal will be written to that location, allowing recovery even if the default temporary directory is lost.

### What happens if the recovery journal is corrupted?

The supervisor validates the journal format before attempting reconstruction. If it encounters an unsupported version or unparseable entry, it aborts recovery and surfaces an error to the user. The session terminates rather than risk undefined behavior from partial state.

### Does replaying the journal duplicate tool calls?

No. The journal uses stable UUIDs for every request and acknowledgment. During reconstruction, the supervisor tracks which operations have been acknowledged. Only unacknowledged requests are reissued to the new worker, preventing duplicate side effects.