How the Prime Agent Daemon Supervisor Monitors and Restarts Workers

The Prime Intellect Agent daemon supervisor keeps session-worker processes alive through PID-based liveness checks, heartbeat snapshots, and automatic restart on failure or timeout.

The daemon supervisor in the Prime Agent repository (PrimeIntellect-ai/prime-agent) is the central process manager responsible for ensuring that worker processes remain healthy and available. It persistently tracks worker state, detects failures through multiple mechanisms, and automatically recovers by relaunching failed workers with their original configuration.

Worker Discovery and Persistence

At startup, the supervisor loads previously saved worker states from JSON descriptor files stored in <agent-dir>/daemon-workers/. This recovery mechanism allows the supervisor to resume monitoring even after a crash or restart.

The key function loadWorkerDescriptors() (≈ line 5200 in daemon-supervisor.ts) scans the descriptor directory and reconstructs the worker registry:

// Pseudo-code based on daemon-supervisor.ts#L5200
const descriptors = await loadWorkerDescriptors(daemonWorkersDir);
for (const desc of descriptors) {
  const worker = await adoptOrRecoverWorker(desc) || await launchWorker(desc);
  this.workers.set(desc.id, worker);
}

Each descriptor contains:

• PID — the operating system process ID
• Socket path — Unix domain socket for command communication
• Recovery journal — crash history for debugging and backoff logic

The supervisor first attempts to adopt an existing worker via adoptOrRecoverWorker(). If the PID is no longer alive, it falls back to launchWorker() to spawn a fresh process with the same session configuration.

Liveness and Heartbeat Monitoring

The daemon supervisor employs two complementary monitoring mechanisms: cheap PID checks and application-level heartbeats.

Process-Level Liveness Checks

The function isProcessAlive() from utils/child-process.ts performs a zero-signal kill test:

// From utils/child-process.ts#L30
export async function isProcessAlive(pid: number): Promise<boolean> {
  try {
    process.kill(pid, 0);
    return true;
  } catch (e) {
    return false;
  }
}

This kill(pid, 0) pattern checks whether the process exists and whether the calling process has permission to signal it, without actually sending a signal.

Application-Level Heartbeat Tracking

Workers periodically emit heartbeat_update commands through the daemon protocol. The supervisor maintains a heartbeatSnapshot (array of AgentConnectionHeartbeat objects) for each worker.

The refreshWorkerSummaries() function (≈ line 7400 in daemon-supervisor.ts) aggregates these snapshots and flags workers as stale when heartbeats stop arriving within WORKER_CONNECT_TIMEOUT_MS (30 seconds).

Failure Detection and Automatic Restart

When the supervisor determines that a worker has failed, it executes a structured recovery sequence.

Failure Conditions

A worker is considered failed when any of the following occur:

• PID check fails — isProcessAlive() returns false
• Request timeout — WORKER_REQUEST_TIMEOUT_MS (approximately 24 hours) exceeded
• Heartbeat stale — no heartbeat_update received within configured window
• Explicit error — DaemonWorkerClient.requestWorker() throws

The Restart Flow

Upon failure detection, the supervisor invokes stopWorker() (≈ line 8600) followed by launchWorker() (≈ line 8800):

// Simplified flow based on daemon-supervisor.ts#L8600-L8800
async handleWorkerFailure(worker: ResidentWorker) {
  await this.stopWorker(worker, /* intentional */ false);
  // Cleanup complete, now respawn
  const newWorker = await this.launchWorker(worker.descriptor);
  await this.persistWorker(newWorker);
}

stopWorker() performs:

1. Sets the mutation drain latch (this.mutationDrain) to block new mutating commands
2. Sends kill command to the worker process
3. Waits for process exit via processIdExists() polling
4. Cleans up socket files and descriptor
5. Records failure in the WorkerRecoveryJournal

launchWorker() then:

1. Spawns a new Node.js process with spawn("node", [...])
2. Creates fresh DaemonWorkerClient for socket communication
3. Re-attaches to existing sessions using the snapshot cache
4. Writes atomic descriptor via persistWorker()

Idle Worker Eviction

Beyond crash recovery, the supervisor proactively manages resource usage through idle eviction. The runIdleEvictionSweep() function (≈ line 7900) runs at intervals controlled by the user setting idleEvictionMinutes.

// Based on daemon-supervisor.ts#L7900
async runIdleEvictionSweep() {
  await this.refreshWorkerSummaries();
  for (const worker of this.workers.values()) {
    if (await this.canEvictWorker(worker)) {
      await worker.client.request({ type: "worker_passivate_idle_children" });
      await this.stopWorker(worker, /* intentional */ true);
    }
  }
}

Evicted workers are intentionally stopped (not crashed), so they are not automatically restarted unless configured to do so.

Daemon Protocol and Communication

All supervisor-worker interaction flows through the daemon protocol defined in daemon-protocol.ts. Key commands include:

Command	Direction	Purpose
heartbeat_update	Worker → Supervisor	Report liveness and connection status
worker_passivate_idle_children	Supervisor → Worker	Request cleanup of idle sessions
kill	Supervisor → Worker	Graceful shutdown request

The DaemonWorkerClient class (daemon-worker-client.ts) wraps socket communication, providing typed request/response handling and timeout management.

Summary

• Persistent state — Worker descriptors in <agent-dir>/daemon-workers/ enable recovery across supervisor restarts
• Dual monitoring — PID checks (isProcessAlive) plus heartbeat snapshots provide reliable failure detection
• Structured restart — stopWorker() cleans up failed workers; launchWorker() spawns replacements with identical configuration
• Atomic persistence — persistWorker() uses write-then-rename for crash-safe descriptor updates
• Resource management — Idle eviction sweeps reclaim memory from inactive workers while preserving session state

Frequently Asked Questions

How does the daemon supervisor detect that a worker process has crashed?

The supervisor detects crashes through PID-based liveness checks using isProcessAlive() from utils/child-process.ts. This function calls process.kill(pid, 0) to verify process existence without sending an actual signal. Additionally, the supervisor monitors heartbeat snapshots that workers update via heartbeat_update commands; missing heartbeats within WORKER_CONNECT_TIMEOUT_MS (30 seconds) flag a worker as stale.

What happens to active sessions when a worker is restarted?

Active sessions are re-attached via the snapshot cache. Before stopping a worker, the supervisor captures the current heartbeatSnapshot containing all AgentConnectionHeartbeat objects. When launchWorker() spawns the replacement process, it uses this snapshot to restore connections to existing sessions, ensuring continuity without client-visible interruption.

What is the worker recovery journal and where is it used?

The worker recovery journal (WorkerRecoveryJournal in worker-recovery-journal.ts) is a persistent log that records worker crashes and recovery attempts. It is updated during stopWorker() when a failure is detected and read during adoptOrRecoverWorker() to implement backoff strategies and provide debugging information for repeated crash loops.

Can the supervisor run multiple instances managing the same workers?

No — the daemon-supervisor-ownership.ts module implements singleton ownership guarantees. It uses file locking and generation counters to ensure only one supervisor instance owns a given socket path. Stale generation detection prevents race conditions where a new supervisor might conflict with a zombie instance.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}