How to Debug Daemon and Worker Issues in Prime Agent Using Logs and Diagnostics

Debug Prime Agent daemon and worker failures by setting PI_LOG_LEVEL=debug, monitoring logs/daemon.log and worker crash files, querying real-time diagnostics via prime-agent daemon-ps --diagnostics, and inspecting the worker recovery journal for incomplete RPCs.

Prime Agent runs its interactive LLM agent inside a daemon process that spawns one or more worker processes for concurrent execution. When debugging daemon and worker issues in Prime Agent, developers rely on configurable logging levels, heartbeat monitoring, and built-in diagnostic RPCs to identify crashes, socket failures, and recovery states. This guide walks through the exact source locations and CLI commands you need to isolate production issues.

Locating and Configuring Prime Agent Logs

Default Log Directory Structure

By default, the daemon writes log files into an logs/ directory adjacent to the agent configuration or next to the Unix socket specified via --daemon-socket. The path resolution logic resides in packages/coding-agent/src/utils/daemon-socket-path.ts, which determines whether logs live in the agent directory or a custom location.

When starting the daemon with a specific socket path, logs co-locate with that socket:

prime-agent --mode daemon --daemon-socket /tmp/prime-agent.sock

# Logs appear in /tmp/logs/ or the configured agent directory

Enabling Verbose Debug Output

The logging system respects the PI_LOG_LEVEL environment variable. Set this to debug or trace before daemon startup to capture granular detail about worker spawns and RPC handling.

export PI_LOG_LEVEL=debug
prime-agent --mode daemon

The implementation in packages/coding-agent/src/core/logging.ts reads this variable and configures the global logger accordingly. You can also adjust this programmatically:

import { setLogLevel } from "packages/coding-agent/src/core/logging.js";

setLogLevel("debug"); // Equivalent to PI_LOG_LEVEL=debug

Debugging Daemon Startup and Connectivity

Startup Sequence Validation

To verify that the daemon can bind its Unix socket and fork worker processes, run the benchmark script scripts/bench-daemon-startup.mjs. This utility spawns a temporary daemon, polls the socket until ready, and prints a concise timeline of initialization events.

node scripts/bench-daemon-startup.mjs

If the daemon fails to start, this script surfaces socket permission errors or port conflicts before you attach production workloads.

Socket Health and Heartbeat Monitoring

The DaemonSupervisor class in packages/coding-agent/src/modes/daemon/daemon-supervisor.ts implements a heartbeat mechanism that logs a tick every DAEMON_HEARTBEAT_MS milliseconds. Gaps in this heartbeat log indicate the daemon is stuck or the socket is blocked.

Monitor the daemon log for heartbeat messages:

tail -f $(prime-agent --config get log-dir)/daemon.log | grep "heartbeat tick"

Missing ticks longer than the configured interval suggest event-loop blocking or resource starvation.

Diagnosing Worker Process Failures

Worker Lifecycle Events

Workers inherit the same logger configuration as the daemon, prefixing all messages with worker:. The supervisor logs every fork, exit, and signal event. When a worker dies unexpectedly, you will see a worker exited line containing the exit code and signal number.

tail -f $(prime-agent --config get log-dir)/daemon.log | grep "worker exited"

These events originate from the handleWorkerExit method in packages/coding-agent/src/modes/daemon/daemon-supervisor.ts, which the supervisor invokes whenever a child process terminates.

Capturing Crash Dumps

When a worker crashes, the supervisor writes a dedicated crash file named worker-<pid>.log inside the logs directory. This file contains the stack trace and the last lines of STDOUT/STDERR captured at termination.

Locate recent crashes:

ls -lt $(prime-agent --config get log-dir)/worker-*.log | head -n 5

The handleWorkerExit routine creates these files automatically, ensuring you have diagnostic data even for segfaults or OOM kills that bypass standard logging.

Using the Diagnostics API and CLI

Real-time Diagnostics with daemon-ps

The supervisor exposes a diagnostics() method that returns a JSON snapshot of current worker health, pending RPCs, and internal counters. Access this via the CLI:

prime-agent daemon-ps --diagnostics

The output includes:

• workers: Array of active worker PIDs and their current task
• pendingRpcCount: Number of requests awaiting worker assignment
• heartbeatMisses: Consecutive missed heartbeats since last check
• lastError: Timestamp and message of the most recent logged error

The CLI implementation resides in packages/coding-agent/src/cli/daemon-ps.ts, which parses the JSON payload for human-readable display.

Programmatic Diagnostics with DaemonClient

For automated health checks, use the DaemonClient class in packages/coding-agent/src/modes/daemon/daemon-client.ts to send a daemon_diagnostics RPC directly:

import { DaemonClient } from "packages/coding-agent/src/modes/daemon/daemon-client.js";

async function getHealthSnapshot(socketPath: string) {
  const client = new DaemonClient(socketPath);
  await client.connect();
  const diag = await client.sendRequest({ type: "daemon_diagnostics" });
  await client.disconnect();
  return diag;
}

// Usage
getHealthSnapshot("/tmp/prime-agent.sock").then(console.log);

This returns the same payload as the CLI, enabling integration with monitoring systems like Prometheus or Datadog.

Recovering from Daemon Restarts

Worker Recovery Journal

If the daemon restarts while workers are processing requests, the WorkerRecoveryJournal in packages/coding-agent/src/modes/daemon/worker-recovery-journal.ts persists incomplete RPC frames to disk. This journal lives under logs/recovery/ and allows the new daemon process to resume or abort in-flight tasks cleanly.

Inspect the recovery directory after an unclean shutdown:

ls $(prime-agent --config get log-dir)/recovery/

Each file represents a serialized RPC frame that was active when the previous daemon process terminated, preventing data loss for long-running LLM inference tasks.

Summary

• Debug logging: Set PI_LOG_LEVEL=debug to enable verbose output in packages/coding-agent/src/core/logging.ts.
• Log locations: Daemon logs reside in the logs/ directory determined by packages/coding-agent/src/utils/daemon-socket-path.ts, with worker crash files following the worker-<pid>.log naming convention.
• Diagnostics: Query live state via prime-agent daemon-ps --diagnostics or programmatically through DaemonClient in packages/coding-agent/src/modes/daemon/daemon-client.ts.
• Crash analysis: Examine handleWorkerExit outputs and heartbeat gaps in packages/coding-agent/src/modes/daemon/daemon-supervisor.ts to identify worker death causes.
• Recovery: Check logs/recovery/ for the WorkerRecoveryJournal data that survives daemon restarts.

Frequently Asked Questions

How do I enable debug logging for Prime Agent daemon processes?

Set the environment variable PI_LOG_LEVEL=debug (or trace for maximum verbosity) before launching the daemon. The logger defined in packages/coding-agent/src/core/logging.ts reads this variable at startup and applies the level globally to both daemon and worker processes.

Where are worker crash logs stored when a Prime Agent worker dies?

Crash logs are written to worker-<pid>.log files inside the configured log directory, typically found via prime-agent --config get log-dir. The handleWorkerExit method in packages/coding-agent/src/modes/daemon/daemon-supervisor.ts generates these files automatically upon detecting a worker exit, capturing the final stack trace and STDOUT/STDERR buffers.

How can I check if the Prime Agent daemon is healthy programmatically?

Import DaemonClient from packages/coding-agent/src/modes/daemon/daemon-client.ts and send a daemon_diagnostics RPC request. This returns a JSON object containing pendingRpcCount, heartbeatMisses, and worker status arrays, allowing you to alert on high queue depths or missed heartbeats.

What happens to incomplete requests when the Prime Agent daemon restarts?

The WorkerRecoveryJournal in packages/coding-agent/src/modes/daemon/worker-recovery-journal.ts serialize incomplete RPC frames to logs/recovery/ before shutdown. When a new daemon starts, it reads these files to resume or cleanly abort the interrupted tasks, preventing data loss for long-running operations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}