# How to Debug Daemon and Worker Issues in Prime Agent Using Logs and Diagnostics

> Troubleshoot Prime Agent daemon and worker issues effectively. Learn to debug failures using logs, diagnostics, and the worker recovery journal.

- Repository: [Prime Intellect/prime-agent](https://github.com/PrimeIntellect-ai/prime-agent)
- Tags: how-to-guide
- Published: 2026-08-18

---

**Debug Prime Agent daemon and worker failures by setting `PI_LOG_LEVEL=debug`, monitoring `logs/daemon.log` and worker crash files, querying real-time diagnostics via `prime-agent daemon-ps --diagnostics`, and inspecting the worker recovery journal for incomplete RPCs.**

Prime Agent runs its interactive LLM agent inside a **daemon process** that spawns one or more **worker processes** for concurrent execution. When debugging daemon and worker issues in Prime Agent, developers rely on configurable logging levels, heartbeat monitoring, and built-in diagnostic RPCs to identify crashes, socket failures, and recovery states. This guide walks through the exact source locations and CLI commands you need to isolate production issues.

## Locating and Configuring Prime Agent Logs

### Default Log Directory Structure

By default, the daemon writes log files into an `logs/` directory adjacent to the agent configuration or next to the Unix socket specified via `--daemon-socket`. The path resolution logic resides in [`packages/coding-agent/src/utils/daemon-socket-path.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/packages/coding-agent/src/utils/daemon-socket-path.ts), which determines whether logs live in the agent directory or a custom location.

When starting the daemon with a specific socket path, logs co-locate with that socket:

```bash
prime-agent --mode daemon --daemon-socket /tmp/prime-agent.sock

# Logs appear in /tmp/logs/ or the configured agent directory

```

### Enabling Verbose Debug Output

The logging system respects the `PI_LOG_LEVEL` environment variable. Set this to `debug` or `trace` before daemon startup to capture granular detail about worker spawns and RPC handling.

```bash
export PI_LOG_LEVEL=debug
prime-agent --mode daemon

```

The implementation in [`packages/coding-agent/src/core/logging.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/packages/coding-agent/src/core/logging.ts) reads this variable and configures the global logger accordingly. You can also adjust this programmatically:

```typescript
import { setLogLevel } from "packages/coding-agent/src/core/logging.js";

setLogLevel("debug"); // Equivalent to PI_LOG_LEVEL=debug

```

## Debugging Daemon Startup and Connectivity

### Startup Sequence Validation

To verify that the daemon can bind its Unix socket and fork worker processes, run the benchmark script `scripts/bench-daemon-startup.mjs`. This utility spawns a temporary daemon, polls the socket until ready, and prints a concise timeline of initialization events.

```bash
node scripts/bench-daemon-startup.mjs

```

If the daemon fails to start, this script surfaces socket permission errors or port conflicts before you attach production workloads.

### Socket Health and Heartbeat Monitoring

The `DaemonSupervisor` class in [`packages/coding-agent/src/modes/daemon/daemon-supervisor.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/packages/coding-agent/src/modes/daemon/daemon-supervisor.ts) implements a heartbeat mechanism that logs a tick every `DAEMON_HEARTBEAT_MS` milliseconds. Gaps in this heartbeat log indicate the daemon is stuck or the socket is blocked.

Monitor the daemon log for heartbeat messages:

```bash
tail -f $(prime-agent --config get log-dir)/daemon.log | grep "heartbeat tick"

```

Missing ticks longer than the configured interval suggest event-loop blocking or resource starvation.

## Diagnosing Worker Process Failures

### Worker Lifecycle Events

Workers inherit the same logger configuration as the daemon, prefixing all messages with `worker:`. The supervisor logs every fork, exit, and signal event. When a worker dies unexpectedly, you will see a `worker exited` line containing the **exit code** and **signal number**.

```bash
tail -f $(prime-agent --config get log-dir)/daemon.log | grep "worker exited"

```

These events originate from the `handleWorkerExit` method in [`packages/coding-agent/src/modes/daemon/daemon-supervisor.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/packages/coding-agent/src/modes/daemon/daemon-supervisor.ts), which the supervisor invokes whenever a child process terminates.

### Capturing Crash Dumps

When a worker crashes, the supervisor writes a dedicated crash file named `worker-<pid>.log` inside the logs directory. This file contains the stack trace and the last lines of STDOUT/STDERR captured at termination.

Locate recent crashes:

```bash
ls -lt $(prime-agent --config get log-dir)/worker-*.log | head -n 5

```

The `handleWorkerExit` routine creates these files automatically, ensuring you have diagnostic data even for segfaults or OOM kills that bypass standard logging.

## Using the Diagnostics API and CLI

### Real-time Diagnostics with daemon-ps

The supervisor exposes a `diagnostics()` method that returns a JSON snapshot of current worker health, pending RPCs, and internal counters. Access this via the CLI:

```bash
prime-agent daemon-ps --diagnostics

```

The output includes:

- **workers**: Array of active worker PIDs and their current task
- **pendingRpcCount**: Number of requests awaiting worker assignment
- **heartbeatMisses**: Consecutive missed heartbeats since last check
- **lastError**: Timestamp and message of the most recent logged error

The CLI implementation resides in [`packages/coding-agent/src/cli/daemon-ps.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/packages/coding-agent/src/cli/daemon-ps.ts), which parses the JSON payload for human-readable display.

### Programmatic Diagnostics with DaemonClient

For automated health checks, use the `DaemonClient` class in [`packages/coding-agent/src/modes/daemon/daemon-client.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/packages/coding-agent/src/modes/daemon/daemon-client.ts) to send a `daemon_diagnostics` RPC directly:

```typescript
import { DaemonClient } from "packages/coding-agent/src/modes/daemon/daemon-client.js";

async function getHealthSnapshot(socketPath: string) {
  const client = new DaemonClient(socketPath);
  await client.connect();
  const diag = await client.sendRequest({ type: "daemon_diagnostics" });
  await client.disconnect();
  return diag;
}

// Usage
getHealthSnapshot("/tmp/prime-agent.sock").then(console.log);

```

This returns the same payload as the CLI, enabling integration with monitoring systems like Prometheus or Datadog.

## Recovering from Daemon Restarts

### Worker Recovery Journal

If the daemon restarts while workers are processing requests, the `WorkerRecoveryJournal` in [`packages/coding-agent/src/modes/daemon/worker-recovery-journal.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/packages/coding-agent/src/modes/daemon/worker-recovery-journal.ts) persists incomplete RPC frames to disk. This journal lives under `logs/recovery/` and allows the new daemon process to resume or abort in-flight tasks cleanly.

Inspect the recovery directory after an unclean shutdown:

```bash
ls $(prime-agent --config get log-dir)/recovery/

```

Each file represents a serialized RPC frame that was active when the previous daemon process terminated, preventing data loss for long-running LLM inference tasks.

## Summary

- **Debug logging**: Set `PI_LOG_LEVEL=debug` to enable verbose output in [`packages/coding-agent/src/core/logging.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/packages/coding-agent/src/core/logging.ts).
- **Log locations**: Daemon logs reside in the `logs/` directory determined by [`packages/coding-agent/src/utils/daemon-socket-path.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/packages/coding-agent/src/utils/daemon-socket-path.ts), with worker crash files following the `worker-<pid>.log` naming convention.
- **Diagnostics**: Query live state via `prime-agent daemon-ps --diagnostics` or programmatically through `DaemonClient` in [`packages/coding-agent/src/modes/daemon/daemon-client.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/packages/coding-agent/src/modes/daemon/daemon-client.ts).
- **Crash analysis**: Examine `handleWorkerExit` outputs and heartbeat gaps in [`packages/coding-agent/src/modes/daemon/daemon-supervisor.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/packages/coding-agent/src/modes/daemon/daemon-supervisor.ts) to identify worker death causes.
- **Recovery**: Check `logs/recovery/` for the `WorkerRecoveryJournal` data that survives daemon restarts.

## Frequently Asked Questions

### How do I enable debug logging for Prime Agent daemon processes?

Set the environment variable `PI_LOG_LEVEL=debug` (or `trace` for maximum verbosity) before launching the daemon. The logger defined in [`packages/coding-agent/src/core/logging.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/packages/coding-agent/src/core/logging.ts) reads this variable at startup and applies the level globally to both daemon and worker processes.

### Where are worker crash logs stored when a Prime Agent worker dies?

Crash logs are written to `worker-<pid>.log` files inside the configured log directory, typically found via `prime-agent --config get log-dir`. The `handleWorkerExit` method in [`packages/coding-agent/src/modes/daemon/daemon-supervisor.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/packages/coding-agent/src/modes/daemon/daemon-supervisor.ts) generates these files automatically upon detecting a worker exit, capturing the final stack trace and STDOUT/STDERR buffers.

### How can I check if the Prime Agent daemon is healthy programmatically?

Import `DaemonClient` from [`packages/coding-agent/src/modes/daemon/daemon-client.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/packages/coding-agent/src/modes/daemon/daemon-client.ts) and send a `daemon_diagnostics` RPC request. This returns a JSON object containing `pendingRpcCount`, `heartbeatMisses`, and worker status arrays, allowing you to alert on high queue depths or missed heartbeats.

### What happens to incomplete requests when the Prime Agent daemon restarts?

The `WorkerRecoveryJournal` in [`packages/coding-agent/src/modes/daemon/worker-recovery-journal.ts`](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/packages/coding-agent/src/modes/daemon/worker-recovery-journal.ts) serialize incomplete RPC frames to `logs/recovery/` before shutdown. When a new daemon starts, it reads these files to resume or cleanly abort the interrupted tasks, preventing data loss for long-running operations.