# How Frigate's Watchdog Monitors and Restarts Failed Processes: A Deep Dive

> Discover how Frigate's watchdog monitors and restarts failed processes. Learn how this essential feature ensures system stability and prevents crashes by automatically relaunching services.

- Repository: [Blake Blackshear/frigate](https://github.com/blakeblackshear/frigate)
- Tags: deep-dive
- Published: 2026-05-25

---

**Frigate uses a dedicated `FrigateWatchdog` thread that polls critical subprocesses every 10 seconds, automatically restarting failed services while throttling rapid crash loops to prevent system instability.**

Frigate's video processing pipeline depends on multiple long-running subprocesses for object detection, recording, and embeddings. To ensure high availability without manual intervention, the application implements a robust **watchdog system** that monitors process health and automatically recovers from failures. This article explores the internals of how `FrigateWatchdog` in [`frigate/watchdog.py`](https://github.com/blakeblackshear/frigate/blob/main/frigate/watchdog.py) detects hung detectors, manages process lifecycles, and implements restart throttling according to the blakeblackshear/frigate source code.

## Core Architecture of the Frigate Watchdog

The watchdog is implemented as **`FrigateWatchdog`**, a subclass of `threading.Thread` that runs as a background daemon. `FrigateApp` instantiates and starts this thread during application bootstrap in [`app.py`](https://github.com/blakeblackshear/frigate/blob/main/app.py) lines 77‑85.

The watchdog receives two critical dependencies during initialization:
- A dictionary of detector processes (`self.detectors`) for health monitoring
- A multiprocessing `stop_event` that signals graceful termination

Each supervised service is wrapped in a **`MonitoredProcess`** dataclass defined in [`watchdog.py`](https://github.com/blakeblackshear/frigate/blob/main/watchdog.py) lines 20‑31. This structure stores:
- The current `FrigateProcess` instance
- A zero-argument **factory** lambda to create fresh process instances
- An optional `on_restart` callback to update parent application state
- A bounded deque of timestamps used to throttle rapid restarts

## Registering Processes for Supervision

During startup, `FrigateApp.start_watchdog()` registers long-running services that require automatic recovery. The registration pattern follows this structure:

```python
self.frigate_watchdog.register(
    key, getattr(self, attr), factory, on_restart
)

```

The `register()` method accepts four parameters:
- **`key`** – The string identifier stored in `self.processes` for health tracking
- **`process`** – The existing `FrigateProcess` instance to monitor
- **`factory`** – A callable that returns a new process instance (e.g., `lambda: EmbeddingProcess(config, stop_event)`)
- **`on_restart`** – A callback executed after successful restart to update the parent app's reference (e.g., `lambda proc: setattr(self, "embeddings", proc)`)

This registration occurs for core services including `EmbeddingProcess`, `RecordProcess`, `ReviewSegmentProcess`, and `OutputProcess` in [`app.py`](https://github.com/blakeblackshear/frigate/blob/main/app.py) lines 106‑119.

## Detecting Failed and Hung Detector Processes

The watchdog's main loop in [`watchdog.py`](https://github.com/blakeblackshear/frigate/blob/main/watchdog.py) lines 119‑135 runs every 10 seconds to evaluate detector health through two distinct failure modes:

**Stuck Detection** – If a detector's `detection_start` timestamp is older than 10 seconds, the watchdog assumes the inference loop is frozen and calls `detector.start_or_restart()`:

```python
detection_start = detector.detection_start.value
if detection_start > 0.0 and now - detection_start > 10:
    logger.info("Detection appears to be stuck. Restarting detection process...")
    detector.start_or_restart()

```

**Process Death Detection** – If `detector.detect_process` exists but `is_alive()` returns `False`, the watchdog assumes catastrophic failure and triggers a full application restart via `restart_frigate()`:

```python
elif detector.detect_process and not detector.detect_process.is_alive():
    logger.info("Detection appears to have stopped. Exiting Frigate...")
    restart_frigate()

```

## Restart Throttling and Crash Loop Prevention

To prevent rapid restart loops that could destabilize the system, the watchdog implements time-window throttling in the `MonitoredProcess` class. Each instance maintains a deque of the last restart timestamps with the following constraints defined in [`watchdog.py`](https://github.com/blakeblackshear/frigate/blob/main/watchdog.py) lines 33‑40:
- **`MAX_RESTARTS = 5`**
- **`RESTART_WINDOW_S = 60`**

The `is_restarting_too_fast()` method discards timestamps older than 60 seconds and returns `True` if more than 5 restarts occurred within the window. When throttled, the watchdog logs the condition and skips the restart attempt, allowing the process to remain dead rather than entering a crash loop.

## The Process Restart Workflow

When `_check_process()` detects a monitored process has exited with a non-zero code, it executes a six-step recovery protocol in [`watchdog.py`](https://github.com/blakeblackshear/frigate/blob/main/watchdog.py) lines 70‑110:

1. **Log the failure** – Records the PID and exit code for debugging.
2. **Throttle check** – Aborts if `is_restarting_too_fast()` returns `True`.
3. **Close the old process** – Calls `entry.process.close()` to release resources.
4. **Create a fresh instance** – Invokes the stored factory: `new_process = entry.factory()`.
5. **Start the new process** – Calls `new_process.start()` to spawn the replacement.
6. **Update bookkeeping** – Replaces the old reference, appends the timestamp to the deque, invokes the `on_restart` callback if provided, and logs success.

If any step raises an exception, the watchdog catches and logs the traceback but continues monitoring other services rather than crashing the entire watchdog thread.

## Graceful Shutdown Behavior

When the application receives a termination signal, `FrigateApp` sets the `stop_event`. The watchdog's main loop checks this event every iteration and performs an orderly exit:
- Breaks the infinite loop when `self.stop_event.is_set()` returns `True`
- Logs "Exiting watchdog…" for observability
- The thread joins back to the main process in `FrigateApp.stop()` at lines 99‑101 of [`app.py`](https://github.com/blakeblackshear/frigate/blob/main/app.py)

This ensures no zombie processes remain and all monitored services receive proper termination signals.

## Practical Implementation Examples

### Registering a Custom Process with the Watchdog

To add your own supervised process, follow the pattern used by `FrigateApp`:

```python
from frigate.util.process import FrigateProcess
from frigate.watchdog import FrigateWatchdog

class MyProcess(FrigateProcess):
    def run(self):
        while not self.stop_event.is_set():
            # Custom processing logic

            pass

def my_factory() -> FrigateProcess:
    return MyProcess(config, stop_event)

watchdog = FrigateWatchdog(detectors, stop_event)

watchdog.register(
    name="my_custom_service",
    process=my_existing_process,
    factory=my_factory,
    on_restart=lambda proc: setattr(app, "my_custom_service", proc),
)

watchdog.start()

```

### Detecting Stuck Detectors in Real-Time

The following snippet demonstrates the exact logic the watchdog uses to evaluate detector health, taken from [`watchdog.py`](https://github.com/blakeblackshear/frigate/blob/main/watchdog.py):

```python

# Inside FrigateWatchdog.run()

for detector in self.detectors.values():
    detection_start = detector.detection_start.value
    if detection_start > 0.0 and now - detection_start > 10:
        logger.info("Detection appears to be stuck. Restarting detection process...")
        detector.start_or_restart()
    elif detector.detect_process and not detector.detect_process.is_alive():
        logger.info("Detection appears to have stopped. Exiting Frigate...")
        restart_frigate()

```

## Summary

- **`FrigateWatchdog`** is a `threading.Thread` subclass that polls every 10 seconds to ensure critical subprocesses remain healthy.
- The **`MonitoredProcess`** dataclass encapsulates process state, factory methods, and restart throttling metadata.
- The watchdog detects **stuck detectors** via `detection_start` timestamps and **dead processes** via `is_alive()` checks.
- **Restart throttling** limits processes to 5 restarts per 60 seconds to prevent system instability from crash loops.
- **Graceful shutdown** is coordinated through a shared `stop_event` that cleanly terminates the monitoring loop.

## Frequently Asked Questions

### How often does Frigate's watchdog check process health?

The watchdog thread sleeps for 10 seconds between iterations, meaning it evaluates detector processes and monitored services every 10 seconds. This interval is hardcoded in the `run()` method of [`frigate/watchdog.py`](https://github.com/blakeblackshear/frigate/blob/main/frigate/watchdog.py).

### What triggers a full Frigate restart versus just a process restart?

If a detector's `detect_process` is no longer alive (indicating the entire detection process has terminated), the watchdog calls `restart_frigate()` to restart the entire application. However, if only the `detection_start` timestamp indicates a stuck inference (older than 10 seconds), the watchdog calls `detector.start_or_restart()` to restart just that specific detector process.

### How does Frigate prevent infinite restart loops?

The watchdog tracks the last 5 restart timestamps in a 60-second window for each monitored process. If `is_restarting_too_fast()` detects more than 5 restarts within 60 seconds, it aborts the restart attempt and logs the error, preventing the system from entering a resource-consuming crash loop.

### Can custom processes be registered with the Frigate watchdog?

Yes. Any class extending `FrigateProcess` can be registered using the `watchdog.register()` method, providing a factory lambda that creates new instances and an optional `on_restart` callback to update the parent application's process references. This pattern mirrors how `FrigateApp` registers built-in services like `EmbeddingProcess` and `RecordProcess`.