# How Claude-Mem Health Monitoring Detects and Recovers From Worker Failures

> Discover how Claude-Mem health monitoring uses PID files, health probes, and version checks to detect and recover from worker failures, ensuring system reliability.

- Repository: [Alex Newman/claude-mem](https://github.com/thedotmack/claude-mem)
- Tags: internals
- Published: 2026-02-16

---

**Claude-Mem continuously monitors worker liveness through PID-file validation, HTTP health probes, and version compatibility checks, automatically spawning replacement processes when failures or mismatches are detected.**

Claude-Mem, an open-source memory system for AI coding assistants maintained at `thedotmack/claude-mem`, relies on a long-lived worker process to host its HTTP API. To ensure high availability for the Claude-Code hook, Cursor integration, and Viewer UI, the system implements a resilient health monitoring subsystem that detects dead or outdated workers and recovers without manual intervention.

## The Health Monitoring Architecture

The detection-and-recovery loop spans three coordinated modules in the `src/services` directory:

- **[`src/services/infrastructure/HealthMonitor.ts`](https://github.com/thedotmack/claude-mem/blob/main/src/services/infrastructure/HealthMonitor.ts)** – Implements low-level HTTP health checks, version validation, and graceful shutdown requests.
- **[`src/services/worker-service.ts`](https://github.com/thedotmack/claude-mem/blob/main/src/services/worker-service.ts)** – Orchestrates the full lifecycle through the `ensureWorkerStarted()` function, coordinating PID cleanup, health probes, and daemon spawning.
- **[`src/services/infrastructure/ProcessManager.ts`](https://github.com/thedotmack/claude-mem/blob/main/src/services/infrastructure/ProcessManager.ts)** – Manages PID-file persistence, stale process detection, and platform-specific spawn logic.

## Detecting Worker Failures

Claude-Mem employs a three-layer detection strategy to identify unresponsive workers.

### PID-File Sanity Checks

When `ensureWorkerStarted()` initiates, it invokes `cleanStalePidFile()` via the `ProcessManager` module. This function reads the stored PID from `~/.claude-mem/worker.pid` and issues a non-blocking `process.kill(pid, 0)` signal. If the process no longer exists, the PID file is immediately removed, signaling that the worker has died.

```typescript
// Conceptual flow from ProcessManager.ts
await cleanStalePidFile(); // Removes ~/.claude-mem/worker.pid if process is dead

```

### Port Probes

The `isPortInUse(port)` function in [`HealthMonitor.ts`](https://github.com/thedotmack/claude-mem/blob/main/HealthMonitor.ts) performs an HTTP `GET` request to `http://127.0.0.1:<port>/api/health`. A successful `200 OK` response confirms the worker is listening; any connection refusal or timeout indicates the port is free, suggesting the worker has crashed or shut down.

```typescript
// From HealthMonitor.ts
const isHealthy = await isPortInUse(37666); // Returns true if /api/health responds

```

### Health-Loop Verification

When a port appears occupied, `waitForHealth(port, timeout)` enters a polling loop, querying the health endpoint every 500 milliseconds until it receives a `200` response or the timeout expires. This handles race conditions where the process exists but the HTTP server hasn't finished initializing.

## Detecting Version Mismatches

A responsive worker may still be incompatible after a Claude-Mem plugin upgrade. The `checkVersionMatch(port)` function addresses this by comparing the installed plugin version (read from the marketplace [`package.json`](https://github.com/thedotmack/claude-mem/blob/main/package.json) via `getInstalledPluginVersion()`) against the running worker's version (fetched from `GET http://127.0.0.1:<port>/api/version` via `getRunningWorkerVersion()`).

If the versions differ, the system returns `{ matches: false }`, triggering a controlled restart to ensure the worker matches the installed codebase.

## Recovery Mechanisms

When detection layers identify a failure or mismatch, Claude-Mem executes a coordinated recovery sequence.

### Graceful Shutdown and Port Release

Upon version mismatch detection, `ensureWorkerStarted()` calls `httpShutdown(port)` to request a graceful stop via the worker's `/api/admin/shutdown` endpoint, followed by `waitForPortFree(port, timeout)` to ensure the TCP socket is fully released before spawning a replacement.

### Daemon Spawning

The `spawnDaemon(__filename, port)` function in [`ProcessManager.ts`](https://github.com/thedotmack/claude-mem/blob/main/ProcessManager.ts) constructs platform-specific commands to launch a detached worker process. On Windows, it uses PowerShell; on Unix systems, it employs `setsid` or plain detached `spawn` to ensure the worker survives the parent's termination.

### Windows Cooldown Protection

To prevent rapid-fire spawn attempts from creating multiple console windows on Windows, `shouldSkipSpawnOnWindows()` checks for a lock file `.worker-start-attempted` and enforces a 2-minute cooldown period between spawn attempts.

### Post-Spawn Verification

After spawning, `ensureWorkerStarted()` re-enters the health verification loop with an extended timeout (`HOOK_TIMEOUTS.POST_SPAWN_WAIT`). If the new worker responds to health checks, the system clears the spawn lock file via `clearWorkerSpawnAttempted()` and returns success. If health checks fail, the PID file is removed and the function returns `false`, allowing the CLI to report an error and the Claude-Code plugin wrapper to retry on the next hook invocation.

## End-to-End Health Check Flow

The complete detection and recovery sequence follows this simplified flow:

```text
ensureWorkerStarted(port)
 ├─ cleanStalePidFile()                ← remove dead PID file
 ├─ isPortInUse(port) ?
 │   └─ yes → waitForHealth()
 │          ├─ health OK → checkVersionMatch()
 │          │   ├─ match → ready
 │          │   └─ mismatch → httpShutdown() → waitForPortFree()
 │          └─ health timeout → spawnDaemon()
 └─ no  → spawnDaemon()
        └─ waitForHealth() → success ? ready : error

```

## Code Examples

### Programmatic Health Check

Use the same function the CLI relies on to verify worker status:

```typescript
import { ensureWorkerStarted } from './services/worker-service';
import { getWorkerPort } from './shared/worker-utils';

async function verifyWorkerHealth() {
  const port = getWorkerPort();
  const isHealthy = await ensureWorkerStarted(port);
  
  if (isHealthy) {
    console.log('✅ Worker is up-to-date and responsive');
  } else {
    console.error('❌ Worker failed health verification');
  }
}

```

> **Source**: `ensureWorkerStarted` implementation in [`src/services/worker-service.ts`](https://github.com/thedotmack/claude-mem/blob/main/src/services/worker-service.ts).

### Manual Health Endpoint Testing

Test the worker's health and version endpoints directly:

```bash

# Check liveness

curl -s http://127.0.0.1:37666/api/health

# Verify version compatibility

curl -s http://127.0.0.1:37666/api/version

```

> **Source**: Health probe functions in [`src/services/infrastructure/HealthMonitor.ts`](https://github.com/thedotmack/claude-mem/blob/main/src/services/infrastructure/HealthMonitor.ts).

### Forcing a Version-Mismatch Recovery

Simulate the automatic restart logic when versions differ:

```typescript
import { httpShutdown, waitForPortFree } from './services/infrastructure/HealthMonitor';
import { getWorkerPort } from './shared/worker-utils';

async function gracefulRestart() {
  const port = getWorkerPort();
  
  // Request graceful shutdown
  await httpShutdown(port);
  
  // Wait for port release
  await waitForPortFree(port, 10000);
  
  // Next call to ensureWorkerStarted() will spawn fresh instance
}

```

> **Source**: Version-mismatch handling in [`src/services/worker-service.ts`](https://github.com/thedotmack/claude-mem/blob/main/src/services/worker-service.ts).

## Summary

- **Three-layer detection**: Claude-Mem validates worker liveness through PID-file verification, HTTP port probes, and iterative health-loop polling.
- **Version safety**: The system compares the installed plugin version against the running worker's reported version via `checkVersionMatch()`, triggering restarts on mismatches.
- **Automatic recovery**: The `ensureWorkerStarted()` orchestrator handles stale PID cleanup, graceful shutdowns, platform-specific daemon spawning, and post-spawn verification without user intervention.
- **Platform protection**: Windows-specific cooldown logic prevents rapid spawn attempts from creating multiple console windows.

## Frequently Asked Questions

### How does Claude-Mem know if the worker process has crashed?

Claude-Mem checks the PID file stored at `~/.claude-mem/worker.pid` using a non-blocking `process.kill(pid, 0)` signal. If the process no longer exists, `cleanStalePidFile()` removes the stale PID file, signaling that the worker has died and needs restart.

### What happens if the worker is running but unresponsive to HTTP requests?

The `isPortInUse()` function attempts a `GET` request to `/api/health`. If the connection is refused or times out, the system treats the port as free. Subsequently, `ensureWorkerStarted()` will spawn a new daemon, and the old unresponsive process will be replaced once the new worker binds to the port.

### How does the system handle plugin updates without manual restarts?

When `ensureWorkerStarted()` detects a running worker, it calls `checkVersionMatch()` to compare the installed plugin version (from [`package.json`](https://github.com/thedotmack/claude-mem/blob/main/package.json)) against the worker's reported version via `/api/version`. If versions differ, the system sends an `httpShutdown()` request, waits for the port to free with `waitForPortFree()`, and spawns a new worker matching the updated codebase.

### Why does Claude-Mem use a cooldown period on Windows?

To prevent rapid spawn attempts from creating multiple PowerShell console windows when the worker fails to start immediately, `shouldSkipSpawnOnWindows()` checks for a `.worker-start-attempted` lock file and enforces a 2-minute cooldown. This ensures the system doesn't overwhelm the user with console pop-ups during recovery attempts.