# How Frigate Configures and Manages Multiple Object Detectors: TensorRT, OpenVINO, CPU, and ONNX

> Learn how Frigate configures and manages multiple object detectors like TensorRT OpenVINO CPU and ONNX using its polymorphic plugin architecture. Optimize your NVR with Frigate.

- Repository: [Blake Blackshear/frigate](https://github.com/blakeblackshear/frigate)
- Tags: internals
- Published: 2026-05-25

---

**Frigate configures and manages multiple object detectors through a polymorphic plugin architecture that maps YAML declarations to specific backend implementations via a factory registry, isolating each detector in its own process while exposing a unified remote detection interface.**

Frigate is an open-source NVR that configures and manages multiple object detectors like TensorRT, OpenVINO, and ONNX through a flexible plugin system. The architecture decouples detection backends from the camera pipeline by registering detector types in a global factory and wrapping each instance in a dedicated process. This design allows a single Frigate instance to run heterogeneous hardware accelerators side-by-side without interference.

## Declarative Detector Configuration

Frigate uses a top-level `detectors:` map in [`config.yml`](https://github.com/blakeblackshear/frigate/blob/main/config.yml) to declare every detector instance. Each entry specifies a `type` literal that determines which plugin implementation to load. The configuration schema in [`frigate/util/schema.py`](https://github.com/blakeblackshear/frigate/blob/main/frigate/util/schema.py) validates these entries against polymorphic `BaseDetectorConfig` subclasses.

```yaml
detectors:
  cpu:
    type: cpu
    model:
      path: cpu_model.tflite
  onnx:
    type: onnx
    device: AUTO
    model:
      path: yolov5.onnx
  openvino:
    type: openvino
    model:
      path: openvino_model.xml
  tensorrt:
    type: tensorrt
    model:
      path: trt_engine.trt

```

Each detector type corresponds to a specific configuration class (e.g., `ONNXDetectorConfig`, `TensorRTDetectorConfig`) that inherits from `BaseDetectorConfig`.

## Plugin Discovery and Registration

Detector plugins reside under `frigate/detectors/plugins/`. The module [`frigate/detectors/__init__.py`](https://github.com/blakeblackshear/frigate/blob/main/frigate/detectors/__init__.py) imports every plugin and extracts its `type` literal to populate a global `api_types` dictionary. This registry creates a runtime mapping from the string type (e.g., `"onnx"`, `"tensorrt"`) to the concrete detector class.

When Frigate starts, the factory queries this registry to instantiate the correct backend based on the user-defined configuration.

## Factory Pattern and Instantiation

The `create_detector(detector_name, detector_cfg)` function in [`frigate/detectors/__init__.py`](https://github.com/blakeblackshear/frigate/blob/main/frigate/detectors/__init__.py) implements the factory pattern. It looks up the detector class from `api_types` using the configuration's `type` field and returns an initialized instance.

```python
from frigate.detectors import create_detector

# Called during startup for each detector defined in the config

for name, cfg in frigate_config.detectors.items():
    detector = create_detector(name, cfg)
    # Returns ONNXDetector, TensorRtDetector, etc.

```

This factory approach ensures that the rest of the codebase remains agnostic to the specific inference backend.

## Process Isolation and Health Monitoring

Each detector runs inside its own **`ObjectDetectProcess`**, a child `multiprocessing.Process` managed by the watchdog in [`frigate/watchdog.py`](https://github.com/blakeblackshear/frigate/blob/main/frigate/watchdog.py). The watchdog maintains a dictionary `self.detectors` mapping detector names to their process instances.

Process isolation guarantees that a crash in one backend (e.g., a TensorRT engine failure) does not terminate the entire NVR. The watchdog monitors detection start times and automatically restarts stuck or crashed processes based on configurable timeouts.

## Unified Detection Interface

Cameras interact with detectors through `RemoteObjectDetector` defined in [`frigate/object_detection/base.py`](https://github.com/blakeblackshear/frigate/blob/main/frigate/object_detection/base.py). This proxy class forwards NumPy tensors to the detector process via `detect_raw()` and returns post-processed detections.

The camera detection loop in [`frigate/video/detect.py`](https://github.com/blakeblackshear/frigate/blob/main/frigate/video/detect.py) receives a `RemoteObjectDetector` instance and calls `object_detector.detect(tensor_input)` without knowledge of the underlying hardware. This abstraction layer remains consistent across all backends.

## Backend-Specific Implementation Details

### ONNX Runtime

The `ONNXDetector` class in [`frigate/detectors/plugins/onnx.py`](https://github.com/blakeblackshear/frigate/blob/main/frigate/detectors/plugins/onnx.py) loads models using `get_optimized_runner()`, which automatically selects the best execution provider (CUDA, ROCm, OpenVINO, or CPU) based on the `device` configuration field. The implementation performs a warm-up inference to initialize execution contexts and avoid watchdog restarts.

### TensorRT

`TensorRTDetector` in [`frigate/detectors/plugins/tensorrt.py`](https://github.com/blakeblackshear/frigate/blob/main/frigate/detectors/plugins/tensorrt.py) only initializes on Jetson devices (`platform.machine() != "x86_64"`). It loads serialized TensorRT engines, creates CUDA buffers, and executes inference via the TensorRT runtime. On x86_64 platforms, the plugin aborts with a clear error directing users to the ONNX detector.

### OpenVINO

The OpenVINO plugin follows the same structural pattern, reading OpenVINO IR format files (`.xml` and `.bin`) and building an inference graph via `IECore` for execution on CPU, GPU, or VPU devices.

### CPU with TensorFlow Lite

The CPU detector utilizes TensorFlow Lite with optional XNNPACK delegates for optimized pure-CPU inference, providing a baseline backend when hardware acceleration is unavailable.

## Camera-to-Detector Binding

By default, cameras use the first detector defined in the configuration. You can override this per camera using the `detector` field:

```yaml
cameras:
  front:
    detect:
      enabled: true
      fps: 10
      detector: tensorrt
  backyard:
    detect:
      enabled: true
      fps: 5
      detector: onnx

```

This binding occurs during pipeline initialization, where the camera's detection loop receives the appropriate `RemoteObjectDetector` instance.

## Code Examples

**Define heterogeneous detectors in [`config.yml`](https://github.com/blakeblackshear/frigate/blob/main/config.yml):**

```yaml
detectors:
  tensorrt:
    type: tensorrt
    device: 0
    model:
      path: /models/trt_engine.trt
      input_tensor: nhwc
      model_type: yolov5
  onnx:
    type: onnx
    device: AUTO
    model:
      path: /models/yolov5.onnx
      input_tensor: nhwc
      model_type: yolox

```

**Remote detection call from the camera pipeline:**

```python

# Inside frigate/video/detect.py

detections = object_detector.detect(tensor_input)

```

## Summary

- **Declarative configuration** uses a YAML map where each detector specifies a `type` literal validated by [`frigate/util/schema.py`](https://github.com/blakeblackshear/frigate/blob/main/frigate/util/schema.py).
- **Plugin registration** occurs in [`frigate/detectors/__init__.py`](https://github.com/blakeblackshear/frigate/blob/main/frigate/detectors/__init__.py), which populates an `api_types` dictionary mapping types to implementation classes.
- **Factory instantiation** via `create_detector()` decouples configuration from backend specifics.
- **Process isolation** via `ObjectDetectProcess` and the watchdog prevents backend failures from crashing the NVR.
- **Unified interface** through `RemoteObjectDetector` allows the camera pipeline to use any backend interchangeably.
- **Backend optimizations** include automatic execution provider selection for ONNX, Jetson-specific TensorRT loading, and XNNPACK delegates for CPU inference.

## Frequently Asked Questions

### Can I run multiple detector types simultaneously in Frigate?

Yes. Frigate supports running heterogeneous detectors side-by-side. Define multiple entries under the `detectors:` map with different types, then assign specific detectors to cameras using the `detector` configuration field. Each backend runs in an isolated process managed by the watchdog.

### How does Frigate select the execution provider for ONNX models?

The `ONNXDetector` uses `get_optimized_runner()` to automatically select the best available execution provider based on the `device` field in your configuration. Valid options include `CUDA`, `ROCm`, `OpenVINO`, `CPU`, or `AUTO`, which attempts providers in order of typical performance.

### What happens if a detector process crashes?

The watchdog in [`frigate/watchdog.py`](https://github.com/blakeblackshear/frigate/blob/main/frigate/watchdog.py) monitors each `ObjectDetectProcess` for health. If a detector stops responding or crashes, the watchdog terminates the stale process and spawns a replacement automatically. This ensures high availability without manual intervention.

### Is TensorRT supported on x86_64 systems?

No. According to the source code in [`frigate/detectors/plugins/tensorrt.py`](https://github.com/blakeblackshear/frigate/blob/main/frigate/detectors/plugins/tensorrt.py), the TensorRT detector explicitly checks `platform.machine()` and aborts on x86_64 architectures with an error message directing users to use the ONNX detector instead. TensorRT is only supported on Jetson ARM devices.