How Prime Agent Handles Supervisor Replacement in Its Daemon Architecture

Prime Agent uses a durable ownership registry, phased restart handshake, and session snapshot replacement to enable seamless supervisor replacement without disrupting active client sessions.

The Prime Agent daemon architecture centers on a long-lived supervisor process that manages session workers and listens on a Unix socket or Windows named pipe. When this supervisor restarts—whether for software updates, crashes, or intentional upgrades—a new instance must take over without breaking existing conversations. This article examines the exact mechanisms implemented in PrimeIntellect-ai/prime-agent that make this possible.

Ownership Acquisition and Stale Supervisor Detection

Before a supervisor can accept connections, it must prove it is the sole legitimate owner of the socket. This process lives in daemon-supervisor-ownership.ts.

The ownership registry structure

The registry uses a filesystem-based approach under PRIME_AGENT_INTERNAL_DAEMON_SUPERVISOR_REGISTRY_DIR. Each candidate supervisor creates a temporary directory containing an owner record with:

• Process PID
• UUID generation
• Socket path
• Random owner token

Conflict resolution logic

// Conceptual flow from daemon-supervisor-ownership.ts
async acquireDaemonSupervisorOwnership() {
  // 1. Create candidate directory with owner record
  const candidateDir = createCandidateRegistryDir();
  
  // 2. Enumerate existing owners while holding registry lock
  for (const existing of enumerateExistingOwners()) {
    if (sameSocketPath(existing) && processIsAlive(existing)) {
      throw new DaemonSupervisorAlreadyRunningError();
    }
    // 3. Replace stale owners
    if (isStale(existing)) {
      renameToStale(existing);
      scheduleCleanup(existing);
    }
  }
  
  // 4. Become the owner
  renameCandidateToFinal(ownerDir);
}

Stale detection works by comparing the recorded PID against the current process table and verifying the process start ID matches. If either check fails, the old supervisor is considered dead and its registry entry can be safely removed.

Hello protocol for replacement detection

Once ownership is established, the supervisor publishes a daemon_hello message containing:

• supervisorGeneration — monotonic counter for replacement detection
• supervisorPid — current process identifier
• supervisorOwnerToken — random token proving identity
• supervisorSocketPath — where clients should connect

Clients store this information and compare it on reconnection to detect when a replacement has occurred.

Three-Phase Update Restart Protocol

When a supervisor needs to restart gracefully—typically during a software update—it transitions through three distinct phases defined in daemon-supervisor.ts.

Phase 1: Draining

Purpose: Finish in-flight work without accepting new mutations.

• Only commands in UPDATE_RESTART_DRAIN_COMMANDS are accepted
• Read-only operations (list, get_messages) continue normally
• Workers remain active but no new sessions start

Phase 2: Fencing

Purpose: Prevent any state mutation while the new supervisor boots.

• All mutable commands rejected with: Daemon is preparing an update restart
• Existing worker connections maintained
• Snapshot streaming continues for attached clients

Phase 3: Prepared

Purpose: Signal readiness for handoff.

• Only shutdown command accepted
• Old supervisor can exit cleanly
• New supervisor can acquire ownership immediately

// From daemon-supervisor.ts: phase transition handling
handleLine(client, line) {
  const { type } = parseCommand(line);
  
  if (this.updateRestartPhase === "draining") {
    if (!UPDATE_RESTART_DRAIN_COMMANDS.includes(type)) {
      return { error: "Daemon is in update restart drain phase" };
    }
  } else if (this.updateRestartPhase === "fencing") {
    if (isMutableCommand(type)) {
      return { error: "Daemon is preparing an update restart" };
    }
  } else if (this.updateRestartPhase === "prepared" && type !== "shutdown") {
    return { error: "Daemon is prepared for restart - only shutdown allowed" };
  }
  // ... normal command dispatch
}

The prepare_update_restart command (defined in daemon-protocol.ts) initiates this sequence, allowing clients to predict and adapt to upcoming replacement.

Session Replacement and Snapshot Handoff

The most critical aspect of supervisor replacement is preserving active sessions. When a worker restarts, its replacement must seamlessly continue conversations.

Worker-to-supervisor notification

When a worker creates a replacement transcript, it sends:

// Outbound message from worker
{
  type: "session_replaced",
  activeSessionId: "sess-abc-123",
  snapshotFollows: true,
  // ... other metadata
}

The supervisor receives this in handleLine and routes it appropriately.

Supervisor handling in handleSnapshotMessage

Located around line 4300 in daemon-supervisor.ts, the logic distinguishes replacement from other snapshot purposes:

// From daemon-supervisor.ts lines 4301-4305
const replacementSnapshotFollows = snapshotPurpose === "replacement";
const snapshotFollows = snapshotPurpose !== "resync";
const outboundType = outbound.type;

if (outboundType === "session_snapshot" || outboundType === "session_replaced") {
  this.handleSnapshotMessage(client, activeSessionId, outbound, snapshotPurpose);
}

Client catchup queueing

The critical recovery mechanism is queueCatchup(), triggered when snapshotPurpose === "replacement":

// Around lines 4170-4175
if (published && (snapshotPurpose === "replacement" || snapshotPurpose === "catchup")) {
  this.queueCatchup(
    client, 
    activeSessionId, 
    snapshotPurpose === "replacement" ? "replacement" : "resync"
  );
}

This ensures clients receive either:

• Replacement snapshot — delta from where they left off
• Full resync — complete transcript if the client cannot handle chunked snapshots

Special handling for session_replaced close types

Lines 4200-4206 reveal careful worker lifecycle management during replacement:

// From daemon-supervisor.ts lines 4200-4206
if (outboundType === "session_replaced" || outboundType === "session_closed") {
  // Keep the workers until all client(s) have fully cleared their queues for this
  // session, the client may be sending the "cancel_prompt_admission" on this
  // session that is enqueued in the worker itself.
  const [Id, sessionId] = (decodedOutbound?.snapshotFollows 
    ? [sessionIdFromId, sessionId] 
    : [sessionIdFromId, null]) as [...any];
  const [f0, f1] = [decodedOutbound?.type === "session_replaced" ? "replace" : ...];
}

Workers are retained longer than the session object itself because clients may have in-flight cancellation messages still queued in the worker's message buffer.

Cleanup and Final Handoff

When replacement completes, the old supervisor executes a structured shutdown sequence.

Ownership release

DaemonSupervisorOwnership.release() performs:

1. Update phase to "stopping" in registry
2. Close all worker process connections via stopWorker()
3. Remove descriptor files from registry directory
4. Release filesystem locks

Idle eviction safety net

Even if cleanup fails, the ownership registry's stale detection (from Section 1) eventually reclaims orphaned entries when their recorded PIDs no longer exist.

Practical Implementation Examples

Detecting supervisor replacement in a client

import { DaemonClient } from "./daemon-client";

const client = new DaemonClient({ socketPath: "/tmp/prime-agent.sock" });

client.on("daemon_hello", (hello) => {
  const storedGeneration = localStorage.getItem("prime-supervisor-gen");
  
  if (storedGeneration && parseInt(storedGeneration) > hello.supervisorGeneration) {
    // Rollback detected - should not happen in normal operation
    console.warn("Supervisor generation regressed");
  } else if (storedGeneration && parseInt(storedGeneration) < hello.supervisorGeneration) {
    // Replacement detected
    console.log(`Supervisor replaced: gen ${storedGeneration} -> ${hello.supervisorGeneration}`);
    reattachAllSessions();
  }
  
  localStorage.setItem("prime-supervisor-gen", String(hello.supervisorGeneration));
  localStorage.setItem("prime-owner-token", hello.supervisorOwnerToken);
});

Initiating graceful replacement for updates

// Update orchestrator script
async function performSupervisorUpdate(newBinaryPath: string) {
  // 1. Signal existing supervisor to start draining
  await client.command({
    type: "prepare_update_restart",
    id: "update-0001",
    targetVersion: "2.0.0"
  });
  
  // 2. Wait for fencing phase (poll status)
  await waitForPhase("fencing");
  
  // 3. Start new supervisor process
  const newSupervisor = spawn(newBinaryPath, [
    "--socket-path", "/tmp/prime-agent.sock",
    "--inherit-registry"
  ]);
  
  // 4. Wait for new supervisor to acquire ownership
  await waitForNewOwnership();
  
  // 5. Confirm old supervisor entered "prepared" phase
  await waitForPhase("prepared");
  
  // 6. Shutdown old supervisor
  await client.command({ type: "shutdown", id: "update-0002" });
}

Handling session replacement mid-conversation

client.onMessage((msg) => {
  switch (msg.type) {
    case "session_replaced":
      // Worker restarted - request continuation snapshot
      client.request({
        type: "attach",
        id: `attach-${Date.now()}`,
        activeSessionId: msg.activeSessionId,
        // Request replacement snapshot specifically
        snapshotPreference: "replacement"
      }).then(response => {
        if (response.snapshot) {
          mergeTranscript(response.snapshot);
        }
      });
      break;
      
    case "session_closed":
      // Permanent closure - save state and notify user
      persistTranscript(msg.finalSnapshot);
      notifyUser("Session ended");
      break;
  }
});

Summary

Prime Agent's supervisor replacement mechanism combines several architectural layers:

• Durable ownership registry — Filesystem-based coordination prevents socket conflicts and enables stale process detection
• Three-phase restart protocol — Draining, fencing, and prepared phases allow graceful degradation of service during handoff
• Session snapshot replacement — Workers generate replacement transcripts that supervisors route to clients via queueCatchup()
• Deferred worker cleanup — session_replaced handling retains workers until client queues drain, preventing message loss

These systems are implemented primarily in daemon-supervisor.ts and daemon-supervisor-ownership.ts, with protocol definitions in daemon-protocol.ts.

Frequently Asked Questions

What happens if the new supervisor crashes during replacement?

If the replacement supervisor fails to fully acquire ownership, the old supervisor remains in the "fencing" or "prepared" phase. Clients continue receiving service from the old supervisor until either the new instance succeeds or an administrator manually aborts the update. The ownership registry's conflict detection prevents a partially-started supervisor from accepting connections.

How does a client know whether to request a replacement snapshot or full resync?

The supervisor determines this via queueCatchup() based on client capabilities negotiated during the attach handshake. If the client advertised chunked snapshot support, it receives a targeted replacement payload. Otherwise, the supervisor sends a complete transcript resynchronization.

Can multiple supervisors run simultaneously for load balancing?

No. The ownership registry enforces single-supervisor semantics per socket path. Attempting to start a second supervisor on the same socket triggers DaemonSupervisorAlreadyRunningError. For multi-supervisor deployments, Prime Agent requires distinct socket paths with external load balancing.

What prevents message loss during the worker replacement window?

The session_replaced handling in lines 4200-4206 deliberately delays worker termination. Even after the session object transitions to replaced status, the underlying worker process continues running until all client message queues are confirmed empty. This ensures in-flight operations like cancel_prompt_admission complete before shutdown.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}