How the no-mistakes Daemon Recovers from Crashes and Manages Orphaned Runs

When the no-mistakes daemon terminates unexpectedly, the next startup detects the stale PID file and automatically executes recoverOnStartup to terminate orphaned child processes, mark interrupted database runs as failed, and remove leftover worktree directories, restoring a consistent state without manual intervention.

The kunchenguid/no-mistakes project implements a resilient daemon architecture designed to withstand sudden crashes from Out-of-Memory kills, panics, or SIGKILL signals. When the daemon process dies abnormally, it leaves behind specific artefacts—PID files, running subprocesses, and active database records—that the next instance must reconcile during startup. This crash recovery mechanism ensures that orphaned runs and resource leaks are automatically cleaned up, preventing the "zombie run" problem common in long-lived CI systems.

Detecting Unexpected Daemon Terminations

The daemon writes its process ID to a PID file at paths.PIDFile during normal startup. If this file exists when the daemon starts, it indicates the previous instance did not shut down gracefully and likely crashed. After completing the recovery routine, the new daemon writes a fresh PID file, preventing race conditions with concurrent starts and clearly delineating the boundaries between daemon instances.

Inside recoverOnStartup: The Four-Stage Recovery Pipeline

Located in internal/daemon/daemon.go at line 45, the recoverOnStartup function orchestrates the comprehensive cleanup process across four distinct domains.

Reaping Orphaned Managed Server Processes

The reapOrphanedServers function in internal/daemon/recover_servers.go (line 17) handles leftover agent processes:

• Agents like Opencode and RovoDev write PID-tracking files when they spawn
• The function compares stored process start times with the current daemon's start time
• It kills any process group whose PID file is stale, preventing leaks of long-running child processes that survived the daemon crash

This ensures that managed servers do not become detached background processes consuming resources indefinitely.

Migrating Legacy Gate Configurations

The migrateGateConfigs function (also in daemon.go around line 300) refreshes post-receive hooks and push-options for legacy bare repositories. While unrelated to process cleanup, this ensures repository-side consistency after an abnormal termination, validating that the Git side of the infrastructure remains synchronized with the daemon state.

Marking Stale Runs as Failed

The database helper d.RecoverStaleRuns("daemon crashed during execution") scans the runs and steps tables for entries that were in_progress during the crash. It updates these rows to a failed status with the supplied error message, ensuring that UI components like axi run display clear failure reasons rather than hanging indefinitely with ambiguous status indicators.

Removing Orphaned Worktree Directories

Each run creates a temporary worktree under $NM_HOME/worktrees/<repo>/<run>. After a crash, these directories may persist as orphaned artefacts. The recovery routine:

• Walks the p.WorktreesDir() directory structure
• Calls git.WorktreeRemove for each orphaned worktree reference
• Falls back to os.RemoveAll on permission or locking errors
• Removes empty repository parent directories
• Logs successful cleanups with structured messages (e.g., "removed orphaned worktree")

Manual Intervention for Runaway Processes

Even after automated cleanup, some runs may persist in the database—particularly CI monitors that continued execution after their worktree was removed. Users can explicitly cancel these orphaned runs using:

no-mistakes axi abort --run <run-id>

This command communicates directly with the daemon via ipc.MethodCancelRun, removing the run from memory without requiring a repository context or local filesystem access.

Simulating and Observing Crash Recovery

To verify recovery behavior in staging environments, operators can simulate a crash and monitor the structured logs:

# Start the daemon (normally automatic when NM_DAEMON=1)

no-mistakes daemon run

# Simulate violent termination (SIGKILL)

kill -9 $(cat "$(no-mistakes info --pid-file)")

# Restart - triggers automatic recovery

no-mistakes daemon run

# Expected log output:

#   "recovered stale runs from previous crash", count=3

#   "removed orphaned worktree", path=/home/user/.nm/worktrees/foo/bar

#   "reaping orphaned managed server", pid=12345

All recovery steps emit structured slog messages, providing operational visibility into which resources were reclaimed and enabling straightforward debugging of crash aftermath.

Summary

• The no-mistakes daemon uses a PID file mechanism at paths.PIDFile to detect unclean shutdowns from crashes, panics, or forced kills
• The recoverOnStartup function in internal/daemon/daemon.go executes a four-stage pipeline: reaping orphaned server processes, migrating gate configs, marking stale runs as failed, and removing orphaned worktree directories
• Database entries in the runs and steps tables are updated with failure status via RecoverStaleRuns to prevent UI hangs and provide clear error attribution
• Users can manually abort persistent orphaned runs using no-mistakes axi abort --run <id> via the ipc.MethodCancelRun IPC method
• All recovery actions emit structured logs for operational visibility and audit trails

Frequently Asked Questions

How does the no-mistakes daemon know a crash occurred?

The daemon checks for the existence of a PID file at paths.PIDFile during startup. If present, it indicates the previous instance did not complete a graceful shutdown, triggering the recoverOnStartup routine before accepting new work.

What happens to active runs when the daemon crashes?

Active runs are marked as failed via d.RecoverStaleRuns("daemon crashed during execution"), which updates the database status for any in-progress entries in the runs and steps tables, ensuring clients see a definitive failure rather than an indefinite hang.

Will orphaned worktree directories consume disk space indefinitely?

No. The recovery routine walks $NM_HOME/worktrees/ and removes orphaned directories using git.WorktreeRemove or os.RemoveAll as a fallback, preventing permanent disk leaks even after severe crashes.

Can I recover from a crashed run without restarting the daemon?

Yes. While automatic recovery requires a daemon restart to trigger recoverOnStartup, you can cancel specific orphaned runs immediately using no-mistakes axi abort --run <run-id>, which invokes ipc.MethodCancelRun to clean up the run in memory without requiring a full daemon restart.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}