# How Openpilot Handles Process Recovery and Automatic Restarts After Crashes

> Discover how openpilot achieves high reliability. Learn about its watchdog system that automatically restarts crashed processes, ensuring continuous operation.

- Repository: [comma.ai/openpilot](https://github.com/commaai/openpilot)
- Tags: internals
- Published: 2026-03-05

---

**Openpilot implements a lightweight internal watchdog in the `system/manager` package that monitors managed daemons on approximately one‑second intervals and immediately relaunches any process configured with `restart_if_crash=True` if it terminates unexpectedly.**

The commaai/openpilot repository relies on a centralized process supervisor to maintain system stability during both on‑road and off‑road operation. Rather than depending on external system‑level init daemons, openpilot embeds its own recovery logic directly into the manager thread, enabling fine‑grained control over which critical components survive unexpected failures.

## The Process Supervision Architecture

At the core of openpilot’s recovery strategy is the **`ManagerProcess`** abstract base class defined in [`system/manager/process.py`](https://github.com/commaai/openpilot/blob/main/system/manager/process.py). Each managed daemon inherits from this class, which exposes a boolean flag **`restart_if_crash`** defaulting to `False`. When enabled, this flag signals the supervisor to resurrect the process immediately after detecting a termination.

The base class provides the **`restart()`** method (lines 70‑83), which forcibly terminates the existing process with `SIGKILL` before invoking `start()` again to spawn a fresh instance:

```python
def restart(self) -> None:
    self.stop(sig=signal.SIGKILL)
    self.start()

```

## Configuring Automatic Restart Behavior

Process declarations reside in [`system/manager/process_config.py`](https://github.com/commaai/openpilot/blob/main/system/manager/process_config.py), where individual entries instantiate subclasses like `PythonProcess` or `NativeProcess`. To enable **automatic restart after crashes**, set `restart_if_crash=True` in the constructor.

Currently, only the UI daemon (`selfdrive.ui.ui`) carries this flag in the master branch (lines 83‑84):

```python
PythonProcess("ui", "selfdrive.ui.ui", always_run, restart_if_crash=True)

```

## The Watchdog Detection Loop

The supervision logic executes inside **`manager_thread()`** within [`system/manager/manager.py`](https://github.com/commaai/openpilot/blob/main/system/manager/manager.py). On each iteration, the thread calls **`ensure_running()`**, passing the complete set of managed processes along with the current vehicle state.

The `ensure_running()` function iterates through every `ManagerProcess` object to verify liveness. When it encounters a process where `restart_if_crash` is `True`, the object exists (`p.proc is not None`), but `p.proc.is_alive()` returns `False`, it logs the exit code via `cloudlog.error` and triggers recovery (lines 55‑60):

```python
if p.restart_if_crash and p.proc is not None and not p.proc.is_alive():
    cloudlog.error(f'Restarting {p.name} (exitcode {p.proc.exitcode})')
    p.restart()

```

After handling crashed processes, the loop starts any newly enabled processes via `p.start()`.

## Shutdown and Cleanup Procedures

When openpilot receives a shutdown request or encounters a fatal error, **`manager_cleanup()`** (lines 95‑103) synchronously terminates all managed processes. Unlike the crash‑recovery path that uses `SIGKILL`, cleanup typically sends standard termination signals and blocks until every process exits, ensuring a clean state before system shutdown.

## Summary

- Openpilot’s **process recovery** is handled internally by the `system/manager` package, requiring no external watchdog daemon.
- The **`restart_if_crash`** flag in [`system/manager/process.py`](https://github.com/commaai/openpilot/blob/main/system/manager/process.py) determines whether a process automatically restarts after failure.
- Only processes explicitly configured in [`system/manager/process_config.py`](https://github.com/commaai/openpilot/blob/main/system/manager/process_config.py) with `restart_if_crash=True` receive automatic recovery; currently only the UI process qualifies.
- The **`ensure_running()`** function checks process liveness on ~1‑second ticks and invokes **`restart()`**, which sends `SIGKILL` before respawning.
- Graceful shutdowns use **`manager_cleanup()`** to stop all processes cleanly without triggering restart logic.

## Frequently Asked Questions

### Which openpilot processes automatically restart after a crash?

According to the current master branch in [`system/manager/process_config.py`](https://github.com/commaai/openpilot/blob/main/system/manager/process_config.py), only the UI daemon (`selfdrive.ui.ui`) has `restart_if_crash` enabled. All other managed processes terminate permanently when they crash and do not respawn automatically.

### How does the manager detect that a process has crashed?

The `ensure_running()` function in [`system/manager/process.py`](https://github.com/commaai/openpilot/blob/main/system/manager/process.py) checks the `is_alive()` method of each process object on every loop iteration, which occurs approximately once per second. If `restart_if_crash` is `True` but `is_alive()` returns `False`, the manager identifies this as a crash and triggers recovery.

### Can I enable automatic restart for custom processes?

Yes. When defining a new process in [`system/manager/process_config.py`](https://github.com/commaai/openpilot/blob/main/system/manager/process_config.py), pass `restart_if_crash=True` to the `PythonProcess` or `NativeProcess` constructor. This instructs the manager to apply the watchdog logic documented in the `ManagerProcess` base class.

### What signal does openpilot use to kill a crashed process during restart?

The `restart()` method in [`system/manager/process.py`](https://github.com/commaai/openpilot/blob/main/system/manager/process.py) explicitly uses `signal.SIGKILL` (via `self.stop(sig=signal.SIGKILL)`) to ensure the crashed process terminates immediately before spawning a replacement, preventing zombie or hung processes from persisting.