How Openpilot Handles Process Recovery and Automatic Restarts After Crashes

Openpilot implements a lightweight internal watchdog in the system/manager package that monitors managed daemons on approximately one‑second intervals and immediately relaunches any process configured with restart_if_crash=True if it terminates unexpectedly.

The commaai/openpilot repository relies on a centralized process supervisor to maintain system stability during both on‑road and off‑road operation. Rather than depending on external system‑level init daemons, openpilot embeds its own recovery logic directly into the manager thread, enabling fine‑grained control over which critical components survive unexpected failures.

The Process Supervision Architecture

At the core of openpilot’s recovery strategy is the ManagerProcess abstract base class defined in system/manager/process.py. Each managed daemon inherits from this class, which exposes a boolean flag restart_if_crash defaulting to False. When enabled, this flag signals the supervisor to resurrect the process immediately after detecting a termination.

The base class provides the restart() method (lines 70‑83), which forcibly terminates the existing process with SIGKILL before invoking start() again to spawn a fresh instance:

def restart(self) -> None:
    self.stop(sig=signal.SIGKILL)
    self.start()

Configuring Automatic Restart Behavior

Process declarations reside in system/manager/process_config.py, where individual entries instantiate subclasses like PythonProcess or NativeProcess. To enable automatic restart after crashes, set restart_if_crash=True in the constructor.

Currently, only the UI daemon (selfdrive.ui.ui) carries this flag in the master branch (lines 83‑84):

PythonProcess("ui", "selfdrive.ui.ui", always_run, restart_if_crash=True)

The Watchdog Detection Loop

The supervision logic executes inside manager_thread() within system/manager/manager.py. On each iteration, the thread calls ensure_running(), passing the complete set of managed processes along with the current vehicle state.

The ensure_running() function iterates through every ManagerProcess object to verify liveness. When it encounters a process where restart_if_crash is True, the object exists (p.proc is not None), but p.proc.is_alive() returns False, it logs the exit code via cloudlog.error and triggers recovery (lines 55‑60):

if p.restart_if_crash and p.proc is not None and not p.proc.is_alive():
    cloudlog.error(f'Restarting {p.name} (exitcode {p.proc.exitcode})')
    p.restart()

After handling crashed processes, the loop starts any newly enabled processes via p.start().

Shutdown and Cleanup Procedures

When openpilot receives a shutdown request or encounters a fatal error, manager_cleanup() (lines 95‑103) synchronously terminates all managed processes. Unlike the crash‑recovery path that uses SIGKILL, cleanup typically sends standard termination signals and blocks until every process exits, ensuring a clean state before system shutdown.

Summary

• Openpilot’s process recovery is handled internally by the system/manager package, requiring no external watchdog daemon.
• The restart_if_crash flag in system/manager/process.py determines whether a process automatically restarts after failure.
• Only processes explicitly configured in system/manager/process_config.py with restart_if_crash=True receive automatic recovery; currently only the UI process qualifies.
• The ensure_running() function checks process liveness on ~1‑second ticks and invokes restart(), which sends SIGKILL before respawning.
• Graceful shutdowns use manager_cleanup() to stop all processes cleanly without triggering restart logic.

Frequently Asked Questions

Which openpilot processes automatically restart after a crash?

According to the current master branch in system/manager/process_config.py, only the UI daemon (selfdrive.ui.ui) has restart_if_crash enabled. All other managed processes terminate permanently when they crash and do not respawn automatically.

How does the manager detect that a process has crashed?

The ensure_running() function in system/manager/process.py checks the is_alive() method of each process object on every loop iteration, which occurs approximately once per second. If restart_if_crash is True but is_alive() returns False, the manager identifies this as a crash and triggers recovery.

Can I enable automatic restart for custom processes?

Yes. When defining a new process in system/manager/process_config.py, pass restart_if_crash=True to the PythonProcess or NativeProcess constructor. This instructs the manager to apply the watchdog logic documented in the ManagerProcess base class.

What signal does openpilot use to kill a crashed process during restart?

The restart() method in system/manager/process.py explicitly uses signal.SIGKILL (via self.stop(sig=signal.SIGKILL)) to ensure the crashed process terminates immediately before spawning a replacement, preventing zombie or hung processes from persisting.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}