# How the no-mistakes Daemon Recovers from Crashes and Manages Orphaned Runs > Learn how the no-mistakes daemon automatically recovers from crashes by terminating orphaned processes, marking failed runs, and cleaning up directories, ensuring a consistent state without manual intervention. - Repository: [Kun Chen/no-mistakes](https://github.com/kunchenguid/no-mistakes) - Tags: internals - Published: 2026-07-04 --- **When the no-mistakes daemon terminates unexpectedly, the next startup detects the stale PID file and automatically executes `recoverOnStartup` to terminate orphaned child processes, mark interrupted database runs as failed, and remove leftover worktree directories, restoring a consistent state without manual intervention.** The kunchenguid/no-mistakes project implements a resilient daemon architecture designed to withstand sudden crashes from Out-of-Memory kills, panics, or `SIGKILL` signals. When the daemon process dies abnormally, it leaves behind specific artefacts—PID files, running subprocesses, and active database records—that the next instance must reconcile during startup. This crash recovery mechanism ensures that orphaned runs and resource leaks are automatically cleaned up, preventing the "zombie run" problem common in long-lived CI systems. ## Detecting Unexpected Daemon Terminations The daemon writes its process ID to a **PID file** at `paths.PIDFile` during normal startup. If this file exists when the daemon starts, it indicates the previous instance did not shut down gracefully and likely crashed. After completing the recovery routine, the new daemon writes a fresh PID file, preventing race conditions with concurrent starts and clearly delineating the boundaries between daemon instances. ## Inside `recoverOnStartup`: The Four-Stage Recovery Pipeline Located in [`internal/daemon/daemon.go`](https://github.com/kunchenguid/no-mistakes/blob/main/internal/daemon/daemon.go) at line 45, the `recoverOnStartup` function orchestrates the comprehensive cleanup process across four distinct domains. ### Reaping Orphaned Managed Server Processes The **`reapOrphanedServers`** function in [`internal/daemon/recover_servers.go`](https://github.com/kunchenguid/no-mistakes/blob/main/internal/daemon/recover_servers.go) (line 17) handles leftover agent processes: - Agents like Opencode and RovoDev write **PID-tracking files** when they spawn - The function compares stored process start times with the current daemon's start time - It kills any process group whose PID file is stale, preventing leaks of long-running child processes that survived the daemon crash This ensures that managed servers do not become detached background processes consuming resources indefinitely. ### Migrating Legacy Gate Configurations The **`migrateGateConfigs`** function (also in [`daemon.go`](https://github.com/kunchenguid/no-mistakes/blob/main/daemon.go) around line 300) refreshes post-receive hooks and push-options for legacy bare repositories. While unrelated to process cleanup, this ensures repository-side consistency after an abnormal termination, validating that the Git side of the infrastructure remains synchronized with the daemon state. ### Marking Stale Runs as Failed The database helper **`d.RecoverStaleRuns("daemon crashed during execution")`** scans the `runs` and `steps` tables for entries that were `in_progress` during the crash. It updates these rows to a **failed status** with the supplied error message, ensuring that UI components like `axi run` display clear failure reasons rather than hanging indefinitely with ambiguous status indicators. ### Removing Orphaned Worktree Directories Each run creates a temporary worktree under `$NM_HOME/worktrees//`. After a crash, these directories may persist as orphaned artefacts. The recovery routine: - Walks the `p.WorktreesDir()` directory structure - Calls `git.WorktreeRemove` for each orphaned worktree reference - Falls back to `os.RemoveAll` on permission or locking errors - Removes empty repository parent directories - Logs successful cleanups with structured messages (e.g., "removed orphaned worktree") ## Manual Intervention for Runaway Processes Even after automated cleanup, some runs may persist in the database—particularly CI monitors that continued execution after their worktree was removed. Users can explicitly cancel these orphaned runs using: ```bash no-mistakes axi abort --run ``` This command communicates directly with the daemon via **`ipc.MethodCancelRun`**, removing the run from memory without requiring a repository context or local filesystem access. ## Simulating and Observing Crash Recovery To verify recovery behavior in staging environments, operators can simulate a crash and monitor the structured logs: ```bash # Start the daemon (normally automatic when NM_DAEMON=1) no-mistakes daemon run # Simulate violent termination (SIGKILL) kill -9 $(cat "$(no-mistakes info --pid-file)") # Restart - triggers automatic recovery no-mistakes daemon run # Expected log output: # "recovered stale runs from previous crash", count=3 # "removed orphaned worktree", path=/home/user/.nm/worktrees/foo/bar # "reaping orphaned managed server", pid=12345 ``` All recovery steps emit structured **slog** messages, providing operational visibility into which resources were reclaimed and enabling straightforward debugging of crash aftermath. ## Summary - The no-mistakes daemon uses a **PID file mechanism** at `paths.PIDFile` to detect unclean shutdowns from crashes, panics, or forced kills - The **`recoverOnStartup`** function in [`internal/daemon/daemon.go`](https://github.com/kunchenguid/no-mistakes/blob/main/internal/daemon/daemon.go) executes a four-stage pipeline: reaping orphaned server processes, migrating gate configs, marking stale runs as failed, and removing orphaned worktree directories - Database entries in the `runs` and `steps` tables are updated with failure status via `RecoverStaleRuns` to prevent UI hangs and provide clear error attribution - Users can manually abort persistent orphaned runs using **`no-mistakes axi abort --run `** via the `ipc.MethodCancelRun` IPC method - All recovery actions emit structured logs for operational visibility and audit trails ## Frequently Asked Questions ### How does the no-mistakes daemon know a crash occurred? The daemon checks for the existence of a PID file at `paths.PIDFile` during startup. If present, it indicates the previous instance did not complete a graceful shutdown, triggering the **`recoverOnStartup`** routine before accepting new work. ### What happens to active runs when the daemon crashes? Active runs are marked as failed via **`d.RecoverStaleRuns("daemon crashed during execution")`**, which updates the database status for any in-progress entries in the `runs` and `steps` tables, ensuring clients see a definitive failure rather than an indefinite hang. ### Will orphaned worktree directories consume disk space indefinitely? No. The recovery routine walks **`$NM_HOME/worktrees/`** and removes orphaned directories using `git.WorktreeRemove` or `os.RemoveAll` as a fallback, preventing permanent disk leaks even after severe crashes. ### Can I recover from a crashed run without restarting the daemon? Yes. While automatic recovery requires a daemon restart to trigger `recoverOnStartup`, you can cancel specific orphaned runs immediately using **`no-mistakes axi abort --run `**, which invokes `ipc.MethodCancelRun` to clean up the run in memory without requiring a full daemon restart.