# How the Prime Agent Daemon Supervisor Monitors and Restarts Workers

> Discover how the Prime Agent daemon supervisor uses PID checks, heartbeats, and automatic restarts to ensure session-worker processes stay active and resilient. Learn more about its monitoring capabilities.

- Repository: [Prime Intellect/prime-agent](https://github.com/PrimeIntellect-ai/prime-agent)
- Tags: internals
- Published: 2026-08-12

---

**The Prime Intellect Agent daemon supervisor keeps session-worker processes alive through PID-based liveness checks, heartbeat snapshots, and automatic restart on failure or timeout.**

The **daemon supervisor** in the Prime Agent repository (`PrimeIntellect-ai/prime-agent`) is the central process manager responsible for ensuring that worker processes remain healthy and available. It persistently tracks worker state, detects failures through multiple mechanisms, and automatically recovers by relaunching failed workers with their original configuration.

## Worker Discovery and Persistence

At startup, the supervisor loads previously saved worker states from JSON descriptor files stored in `<agent-dir>/daemon-workers/`. This recovery mechanism allows the supervisor to resume monitoring even after a crash or restart.

The key function `loadWorkerDescriptors()` (≈ line 5200 in [`daemon-supervisor.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/daemon-supervisor.ts)) scans the descriptor directory and reconstructs the worker registry:

```typescript
// Pseudo-code based on daemon-supervisor.ts#L5200
const descriptors = await loadWorkerDescriptors(daemonWorkersDir);
for (const desc of descriptors) {
  const worker = await adoptOrRecoverWorker(desc) || await launchWorker(desc);
  this.workers.set(desc.id, worker);
}

```

Each descriptor contains:

- **PID** — the operating system process ID
- **Socket path** — Unix domain socket for command communication
- **Recovery journal** — crash history for debugging and backoff logic

The supervisor first attempts to **adopt** an existing worker via `adoptOrRecoverWorker()`. If the PID is no longer alive, it falls back to `launchWorker()` to spawn a fresh process with the same session configuration.

## Liveness and Heartbeat Monitoring

The daemon supervisor employs two complementary monitoring mechanisms: cheap PID checks and application-level heartbeats.

### Process-Level Liveness Checks

The function `isProcessAlive()` from [`utils/child-process.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/utils/child-process.ts) performs a zero-signal kill test:

```typescript
// From utils/child-process.ts#L30
export async function isProcessAlive(pid: number): Promise<boolean> {
  try {
    process.kill(pid, 0);
    return true;
  } catch (e) {
    return false;
  }
}

```

This `kill(pid, 0)` pattern checks whether the process exists and whether the calling process has permission to signal it, without actually sending a signal.

### Application-Level Heartbeat Tracking

Workers periodically emit `heartbeat_update` commands through the daemon protocol. The supervisor maintains a **`heartbeatSnapshot`** (array of `AgentConnectionHeartbeat` objects) for each worker.

The `refreshWorkerSummaries()` function (≈ line 7400 in [`daemon-supervisor.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/daemon-supervisor.ts)) aggregates these snapshots and flags workers as stale when heartbeats stop arriving within `WORKER_CONNECT_TIMEOUT_MS` (30 seconds).

## Failure Detection and Automatic Restart

When the supervisor determines that a worker has failed, it executes a structured recovery sequence.

### Failure Conditions

A worker is considered failed when any of the following occur:

- **PID check fails** — `isProcessAlive()` returns false
- **Request timeout** — `WORKER_REQUEST_TIMEOUT_MS` (approximately 24 hours) exceeded
- **Heartbeat stale** — no `heartbeat_update` received within configured window
- **Explicit error** — `DaemonWorkerClient.requestWorker()` throws

### The Restart Flow

Upon failure detection, the supervisor invokes `stopWorker()` (≈ line 8600) followed by `launchWorker()` (≈ line 8800):

```typescript
// Simplified flow based on daemon-supervisor.ts#L8600-L8800
async handleWorkerFailure(worker: ResidentWorker) {
  await this.stopWorker(worker, /* intentional */ false);
  // Cleanup complete, now respawn
  const newWorker = await this.launchWorker(worker.descriptor);
  await this.persistWorker(newWorker);
}

```

**`stopWorker()`** performs:

1. Sets the **mutation drain latch** (`this.mutationDrain`) to block new mutating commands
2. Sends `kill` command to the worker process
3. Waits for process exit via `processIdExists()` polling
4. Cleans up socket files and descriptor
5. Records failure in the `WorkerRecoveryJournal`

**`launchWorker()`** then:

1. Spawns a new Node.js process with `spawn("node", [...])`
2. Creates fresh `DaemonWorkerClient` for socket communication
3. Re-attaches to existing sessions using the snapshot cache
4. Writes atomic descriptor via `persistWorker()`

## Idle Worker Eviction

Beyond crash recovery, the supervisor proactively manages resource usage through idle eviction. The `runIdleEvictionSweep()` function (≈ line 7900) runs at intervals controlled by the user setting `idleEvictionMinutes`.

```typescript
// Based on daemon-supervisor.ts#L7900
async runIdleEvictionSweep() {
  await this.refreshWorkerSummaries();
  for (const worker of this.workers.values()) {
    if (await this.canEvictWorker(worker)) {
      await worker.client.request({ type: "worker_passivate_idle_children" });
      await this.stopWorker(worker, /* intentional */ true);
    }
  }
}

```

Evicted workers are intentionally stopped (not crashed), so they are not automatically restarted unless configured to do so.

## Daemon Protocol and Communication

All supervisor-worker interaction flows through the **daemon protocol** defined in [`daemon-protocol.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/daemon-protocol.ts). Key commands include:

| Command | Direction | Purpose |
|---------|-----------|---------|
| `heartbeat_update` | Worker → Supervisor | Report liveness and connection status |
| `worker_passivate_idle_children` | Supervisor → Worker | Request cleanup of idle sessions |
| `kill` | Supervisor → Worker | Graceful shutdown request |

The `DaemonWorkerClient` class ([`daemon-worker-client.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/daemon-worker-client.ts)) wraps socket communication, providing typed request/response handling and timeout management.

## Summary

- **Persistent state** — Worker descriptors in `<agent-dir>/daemon-workers/` enable recovery across supervisor restarts
- **Dual monitoring** — PID checks (`isProcessAlive`) plus heartbeat snapshots provide reliable failure detection
- **Structured restart** — `stopWorker()` cleans up failed workers; `launchWorker()` spawns replacements with identical configuration
- **Atomic persistence** — `persistWorker()` uses write-then-rename for crash-safe descriptor updates
- **Resource management** — Idle eviction sweeps reclaim memory from inactive workers while preserving session state

## Frequently Asked Questions

### How does the daemon supervisor detect that a worker process has crashed?

The supervisor detects crashes through **PID-based liveness checks** using `isProcessAlive()` from [`utils/child-process.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/utils/child-process.ts). This function calls `process.kill(pid, 0)` to verify process existence without sending an actual signal. Additionally, the supervisor monitors **heartbeat snapshots** that workers update via `heartbeat_update` commands; missing heartbeats within `WORKER_CONNECT_TIMEOUT_MS` (30 seconds) flag a worker as stale.

### What happens to active sessions when a worker is restarted?

Active sessions are **re-attached via the snapshot cache**. Before stopping a worker, the supervisor captures the current `heartbeatSnapshot` containing all `AgentConnectionHeartbeat` objects. When `launchWorker()` spawns the replacement process, it uses this snapshot to restore connections to existing sessions, ensuring continuity without client-visible interruption.

### What is the worker recovery journal and where is it used?

The **worker recovery journal** (`WorkerRecoveryJournal` in [`worker-recovery-journal.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/worker-recovery-journal.ts)) is a persistent log that records worker crashes and recovery attempts. It is updated during `stopWorker()` when a failure is detected and read during `adoptOrRecoverWorker()` to implement backoff strategies and provide debugging information for repeated crash loops.

### Can the supervisor run multiple instances managing the same workers?

No — the [`daemon-supervisor-ownership.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/daemon-supervisor-ownership.ts) module implements **singleton ownership** guarantees. It uses file locking and generation counters to ensure only one supervisor instance owns a given socket path. Stale generation detection prevents race conditions where a new supervisor might conflict with a zombie instance.