How Frigate Configures and Manages Multiple Object Detectors: TensorRT, OpenVINO, CPU, and ONNX

Frigate configures and manages multiple object detectors through a polymorphic plugin architecture that maps YAML declarations to specific backend implementations via a factory registry, isolating each detector in its own process while exposing a unified remote detection interface.

Frigate is an open-source NVR that configures and manages multiple object detectors like TensorRT, OpenVINO, and ONNX through a flexible plugin system. The architecture decouples detection backends from the camera pipeline by registering detector types in a global factory and wrapping each instance in a dedicated process. This design allows a single Frigate instance to run heterogeneous hardware accelerators side-by-side without interference.

Declarative Detector Configuration

Frigate uses a top-level detectors: map in config.yml to declare every detector instance. Each entry specifies a type literal that determines which plugin implementation to load. The configuration schema in frigate/util/schema.py validates these entries against polymorphic BaseDetectorConfig subclasses.

detectors:
  cpu:
    type: cpu
    model:
      path: cpu_model.tflite
  onnx:
    type: onnx
    device: AUTO
    model:
      path: yolov5.onnx
  openvino:
    type: openvino
    model:
      path: openvino_model.xml
  tensorrt:
    type: tensorrt
    model:
      path: trt_engine.trt

Each detector type corresponds to a specific configuration class (e.g., ONNXDetectorConfig, TensorRTDetectorConfig) that inherits from BaseDetectorConfig.

Plugin Discovery and Registration

Detector plugins reside under frigate/detectors/plugins/. The module frigate/detectors/__init__.py imports every plugin and extracts its type literal to populate a global api_types dictionary. This registry creates a runtime mapping from the string type (e.g., "onnx", "tensorrt") to the concrete detector class.

When Frigate starts, the factory queries this registry to instantiate the correct backend based on the user-defined configuration.

Factory Pattern and Instantiation

The create_detector(detector_name, detector_cfg) function in frigate/detectors/__init__.py implements the factory pattern. It looks up the detector class from api_types using the configuration's type field and returns an initialized instance.

from frigate.detectors import create_detector

# Called during startup for each detector defined in the config

for name, cfg in frigate_config.detectors.items():
    detector = create_detector(name, cfg)
    # Returns ONNXDetector, TensorRtDetector, etc.

This factory approach ensures that the rest of the codebase remains agnostic to the specific inference backend.

Process Isolation and Health Monitoring

Each detector runs inside its own ObjectDetectProcess, a child multiprocessing.Process managed by the watchdog in frigate/watchdog.py. The watchdog maintains a dictionary self.detectors mapping detector names to their process instances.

Process isolation guarantees that a crash in one backend (e.g., a TensorRT engine failure) does not terminate the entire NVR. The watchdog monitors detection start times and automatically restarts stuck or crashed processes based on configurable timeouts.

Unified Detection Interface

Cameras interact with detectors through RemoteObjectDetector defined in frigate/object_detection/base.py. This proxy class forwards NumPy tensors to the detector process via detect_raw() and returns post-processed detections.

The camera detection loop in frigate/video/detect.py receives a RemoteObjectDetector instance and calls object_detector.detect(tensor_input) without knowledge of the underlying hardware. This abstraction layer remains consistent across all backends.

Backend-Specific Implementation Details

ONNX Runtime

The ONNXDetector class in frigate/detectors/plugins/onnx.py loads models using get_optimized_runner(), which automatically selects the best execution provider (CUDA, ROCm, OpenVINO, or CPU) based on the device configuration field. The implementation performs a warm-up inference to initialize execution contexts and avoid watchdog restarts.

TensorRT

TensorRTDetector in frigate/detectors/plugins/tensorrt.py only initializes on Jetson devices (platform.machine() != "x86_64"). It loads serialized TensorRT engines, creates CUDA buffers, and executes inference via the TensorRT runtime. On x86_64 platforms, the plugin aborts with a clear error directing users to the ONNX detector.

OpenVINO

The OpenVINO plugin follows the same structural pattern, reading OpenVINO IR format files (.xml and .bin) and building an inference graph via IECore for execution on CPU, GPU, or VPU devices.

CPU with TensorFlow Lite

The CPU detector utilizes TensorFlow Lite with optional XNNPACK delegates for optimized pure-CPU inference, providing a baseline backend when hardware acceleration is unavailable.

Camera-to-Detector Binding

By default, cameras use the first detector defined in the configuration. You can override this per camera using the detector field:

cameras:
  front:
    detect:
      enabled: true
      fps: 10
      detector: tensorrt
  backyard:
    detect:
      enabled: true
      fps: 5
      detector: onnx

This binding occurs during pipeline initialization, where the camera's detection loop receives the appropriate RemoteObjectDetector instance.

Code Examples

Define heterogeneous detectors in config.yml:

detectors:
  tensorrt:
    type: tensorrt
    device: 0
    model:
      path: /models/trt_engine.trt
      input_tensor: nhwc
      model_type: yolov5
  onnx:
    type: onnx
    device: AUTO
    model:
      path: /models/yolov5.onnx
      input_tensor: nhwc
      model_type: yolox

Remote detection call from the camera pipeline:

# Inside frigate/video/detect.py

detections = object_detector.detect(tensor_input)

Summary

• Declarative configuration uses a YAML map where each detector specifies a type literal validated by frigate/util/schema.py.
• Plugin registration occurs in frigate/detectors/__init__.py, which populates an api_types dictionary mapping types to implementation classes.
• Factory instantiation via create_detector() decouples configuration from backend specifics.
• Process isolation via ObjectDetectProcess and the watchdog prevents backend failures from crashing the NVR.
• Unified interface through RemoteObjectDetector allows the camera pipeline to use any backend interchangeably.
• Backend optimizations include automatic execution provider selection for ONNX, Jetson-specific TensorRT loading, and XNNPACK delegates for CPU inference.

Frequently Asked Questions

Can I run multiple detector types simultaneously in Frigate?

Yes. Frigate supports running heterogeneous detectors side-by-side. Define multiple entries under the detectors: map with different types, then assign specific detectors to cameras using the detector configuration field. Each backend runs in an isolated process managed by the watchdog.

How does Frigate select the execution provider for ONNX models?

The ONNXDetector uses get_optimized_runner() to automatically select the best available execution provider based on the device field in your configuration. Valid options include CUDA, ROCm, OpenVINO, CPU, or AUTO, which attempts providers in order of typical performance.

What happens if a detector process crashes?

The watchdog in frigate/watchdog.py monitors each ObjectDetectProcess for health. If a detector stops responding or crashes, the watchdog terminates the stale process and spawns a replacement automatically. This ensures high availability without manual intervention.

Is TensorRT supported on x86_64 systems?

No. According to the source code in frigate/detectors/plugins/tensorrt.py, the TensorRT detector explicitly checks platform.machine() and aborts on x86_64 architectures with an error message directing users to use the ONNX detector instead. TensorRT is only supported on Jetson ARM devices.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}