How Frigate's Watchdog Monitors and Restarts Failed Processes: A Deep Dive

Frigate uses a dedicated FrigateWatchdog thread that polls critical subprocesses every 10 seconds, automatically restarting failed services while throttling rapid crash loops to prevent system instability.

Frigate's video processing pipeline depends on multiple long-running subprocesses for object detection, recording, and embeddings. To ensure high availability without manual intervention, the application implements a robust watchdog system that monitors process health and automatically recovers from failures. This article explores the internals of how FrigateWatchdog in frigate/watchdog.py detects hung detectors, manages process lifecycles, and implements restart throttling according to the blakeblackshear/frigate source code.

Core Architecture of the Frigate Watchdog

The watchdog is implemented as FrigateWatchdog, a subclass of threading.Thread that runs as a background daemon. FrigateApp instantiates and starts this thread during application bootstrap in app.py lines 77‑85.

The watchdog receives two critical dependencies during initialization:

• A dictionary of detector processes (self.detectors) for health monitoring
• A multiprocessing stop_event that signals graceful termination

Each supervised service is wrapped in a MonitoredProcess dataclass defined in watchdog.py lines 20‑31. This structure stores:

• The current FrigateProcess instance
• A zero-argument factory lambda to create fresh process instances
• An optional on_restart callback to update parent application state
• A bounded deque of timestamps used to throttle rapid restarts

Registering Processes for Supervision

During startup, FrigateApp.start_watchdog() registers long-running services that require automatic recovery. The registration pattern follows this structure:

self.frigate_watchdog.register(
    key, getattr(self, attr), factory, on_restart
)

The register() method accepts four parameters:

• key – The string identifier stored in self.processes for health tracking
• process – The existing FrigateProcess instance to monitor
• factory – A callable that returns a new process instance (e.g., lambda: EmbeddingProcess(config, stop_event))
• on_restart – A callback executed after successful restart to update the parent app's reference (e.g., lambda proc: setattr(self, "embeddings", proc))

This registration occurs for core services including EmbeddingProcess, RecordProcess, ReviewSegmentProcess, and OutputProcess in app.py lines 106‑119.

Detecting Failed and Hung Detector Processes

The watchdog's main loop in watchdog.py lines 119‑135 runs every 10 seconds to evaluate detector health through two distinct failure modes:

Stuck Detection – If a detector's detection_start timestamp is older than 10 seconds, the watchdog assumes the inference loop is frozen and calls detector.start_or_restart():

detection_start = detector.detection_start.value
if detection_start > 0.0 and now - detection_start > 10:
    logger.info("Detection appears to be stuck. Restarting detection process...")
    detector.start_or_restart()

Process Death Detection – If detector.detect_process exists but is_alive() returns False, the watchdog assumes catastrophic failure and triggers a full application restart via restart_frigate():

elif detector.detect_process and not detector.detect_process.is_alive():
    logger.info("Detection appears to have stopped. Exiting Frigate...")
    restart_frigate()

Restart Throttling and Crash Loop Prevention

To prevent rapid restart loops that could destabilize the system, the watchdog implements time-window throttling in the MonitoredProcess class. Each instance maintains a deque of the last restart timestamps with the following constraints defined in watchdog.py lines 33‑40:

• MAX_RESTARTS = 5
• RESTART_WINDOW_S = 60

The is_restarting_too_fast() method discards timestamps older than 60 seconds and returns True if more than 5 restarts occurred within the window. When throttled, the watchdog logs the condition and skips the restart attempt, allowing the process to remain dead rather than entering a crash loop.

The Process Restart Workflow

When _check_process() detects a monitored process has exited with a non-zero code, it executes a six-step recovery protocol in watchdog.py lines 70‑110:

1. Log the failure – Records the PID and exit code for debugging.
2. Throttle check – Aborts if is_restarting_too_fast() returns True.
3. Close the old process – Calls entry.process.close() to release resources.
4. Create a fresh instance – Invokes the stored factory: new_process = entry.factory().
5. Start the new process – Calls new_process.start() to spawn the replacement.
6. Update bookkeeping – Replaces the old reference, appends the timestamp to the deque, invokes the on_restart callback if provided, and logs success.

If any step raises an exception, the watchdog catches and logs the traceback but continues monitoring other services rather than crashing the entire watchdog thread.

Graceful Shutdown Behavior

When the application receives a termination signal, FrigateApp sets the stop_event. The watchdog's main loop checks this event every iteration and performs an orderly exit:

• Breaks the infinite loop when self.stop_event.is_set() returns True
• Logs "Exiting watchdog…" for observability
• The thread joins back to the main process in FrigateApp.stop() at lines 99‑101 of app.py

This ensures no zombie processes remain and all monitored services receive proper termination signals.

Practical Implementation Examples

Registering a Custom Process with the Watchdog

To add your own supervised process, follow the pattern used by FrigateApp:

from frigate.util.process import FrigateProcess
from frigate.watchdog import FrigateWatchdog

class MyProcess(FrigateProcess):
    def run(self):
        while not self.stop_event.is_set():
            # Custom processing logic

            pass

def my_factory() -> FrigateProcess:
    return MyProcess(config, stop_event)

watchdog = FrigateWatchdog(detectors, stop_event)

watchdog.register(
    name="my_custom_service",
    process=my_existing_process,
    factory=my_factory,
    on_restart=lambda proc: setattr(app, "my_custom_service", proc),
)

watchdog.start()

Detecting Stuck Detectors in Real-Time

The following snippet demonstrates the exact logic the watchdog uses to evaluate detector health, taken from watchdog.py:

# Inside FrigateWatchdog.run()

for detector in self.detectors.values():
    detection_start = detector.detection_start.value
    if detection_start > 0.0 and now - detection_start > 10:
        logger.info("Detection appears to be stuck. Restarting detection process...")
        detector.start_or_restart()
    elif detector.detect_process and not detector.detect_process.is_alive():
        logger.info("Detection appears to have stopped. Exiting Frigate...")
        restart_frigate()

Summary

• FrigateWatchdog is a threading.Thread subclass that polls every 10 seconds to ensure critical subprocesses remain healthy.
• The MonitoredProcess dataclass encapsulates process state, factory methods, and restart throttling metadata.
• The watchdog detects stuck detectors via detection_start timestamps and dead processes via is_alive() checks.
• Restart throttling limits processes to 5 restarts per 60 seconds to prevent system instability from crash loops.
• Graceful shutdown is coordinated through a shared stop_event that cleanly terminates the monitoring loop.

Frequently Asked Questions

How often does Frigate's watchdog check process health?

The watchdog thread sleeps for 10 seconds between iterations, meaning it evaluates detector processes and monitored services every 10 seconds. This interval is hardcoded in the run() method of frigate/watchdog.py.

What triggers a full Frigate restart versus just a process restart?

If a detector's detect_process is no longer alive (indicating the entire detection process has terminated), the watchdog calls restart_frigate() to restart the entire application. However, if only the detection_start timestamp indicates a stuck inference (older than 10 seconds), the watchdog calls detector.start_or_restart() to restart just that specific detector process.

How does Frigate prevent infinite restart loops?

The watchdog tracks the last 5 restart timestamps in a 60-second window for each monitored process. If is_restarting_too_fast() detects more than 5 restarts within 60 seconds, it aborts the restart attempt and logs the error, preventing the system from entering a resource-consuming crash loop.

Can custom processes be registered with the Frigate watchdog?

Yes. Any class extending FrigateProcess can be registered using the watchdog.register() method, providing a factory lambda that creates new instances and an optional on_restart callback to update the parent application's process references. This pattern mirrors how FrigateApp registers built-in services like EmbeddingProcess and RecordProcess.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}