# How ONNX Runtime's Execution Provider Interface Works: A Deep Dive into Hardware Abstraction

> Understand ONNX Runtime's Execution Provider interface discover how it abstracts hardware for CPUs GPUs and custom accelerators Learn about plug-in kernels memory allocators and data transfer utilities

- Repository: [Microsoft/onnxruntime](https://github.com/microsoft/onnxruntime)
- Tags: deep-dive
- Published: 2026-04-24

---

**ONNX Runtime's Execution Provider (EP) interface is an abstract plug-in architecture that enables hardware-specific backends to supply kernels, memory allocators, and data-transfer utilities, allowing the framework to offload computation to CPUs, GPUs, and custom accelerators without modifying the core runtime.**

The microsoft/onnxruntime repository implements this sophisticated abstraction layer to decouple the graph execution engine from hardware-specific details. By defining a standardized contract in the core framework headers, the EP interface allows diverse backends to integrate seamlessly while the session orchestrates partitioning, memory management, and kernel dispatch.

## Core Abstractions of the Execution Provider Interface

The EP architecture revolves around several key components that define the contract between the framework and hardware backends.

### IExecutionProvider: The Abstract Base Class

Every hardware backend derives from **IExecutionProvider**, defined in [`include/onnxruntime/core/framework/execution_provider.h`](https://github.com/microsoft/onnxruntime/blob/main/include/onnxruntime/core/framework/execution_provider.h) (lines 74-155). This abstract base class declares the essential virtual methods that concrete implementations must override:

- **`GetCapability`** – Inspects the graph and returns a vector of `ComputeCapability` objects describing which sub-graphs the EP can execute.
- **`GetKernelRegistry`** – Returns a `std::shared_ptr<KernelRegistry>` containing the kernel factories (`KernelCreateInfo`) for supported operators.
- **`CreatePreferredAllocators`** – Supplies device-specific memory allocators for tensor storage.
- **`GetDataTransfer`** – Returns an `IDataTransfer` instance for moving tensors between host and device memory.

The base class also defines lifecycle callbacks including `OnRunStart`, `OnRunEnd`, and `Sync`, which the session invokes during inference.

### ExecutionProviders Container

The **`ExecutionProviders`** class, located in [`onnxruntime/core/framework/execution_providers.h`](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/core/framework/execution_providers.h) (lines 24-84), manages the collection of EP instances attached to an `InferenceSession`. It provides fast lookup by provider name and stores provider-specific options through the `Add` method, which registers each EP shared pointer with the session.

### KernelRegistry and ComputeCapability

When the session partitions the graph, it queries each EP's capability through **ComputeCapability** objects. These structures describe executable sub-graphs, whether single nodes or fused groups. The EP populates these by implementing `GetCapability` and inspecting the `GraphViewer` to verify supported operations via the supplied **`IKernelLookup`** interface.

The **`KernelRegistry`** holds the mapping between ONNX operator types and their concrete `OpKernel` implementations. When a node is assigned to an EP, the session calls `GetKernelRegistry` to retrieve the factory that instantiates the device-specific kernel.

### Memory and Data Transfer Interfaces

EPs expose hardware resources through **OrtDevice** and **OrtEpDevice** descriptors, which include device IDs, types, and auxiliary devices for multi-GPU configurations. The **`CreatePreferredAllocators`** method allows EPs to override default memory allocation strategies, while **`GetDataTransfer`** enables custom copy routines between host and device buffers.

## Execution Provider Lifecycle in ONNX Runtime

The EP passes through distinct phases during session initialization and runtime execution.

### Construction and Registration

Concrete EP implementations, such as **`CPUExecutionProvider`** in `onnxruntime/core/providers/cpu/cpu_execution_provider.cc`, derive from `IExecutionProvider` and supply a type string (e.g., `kCpuExecutionProvider`) in their constructor:

```cpp
CPUExecutionProvider(const CPUExecutionProviderInfo& info)
    : IExecutionProvider{onnxruntime::kCpuExecutionProvider}, info_{info} {}

```

The application registers EPs through `ExecutionProviders::Add` or via the C++ API:

```cpp
Ort::SessionOptions session_options;
session_options.AppendExecutionProvider_CPU(Ort::AllocatorWithDefaultOptions());

```

### Capability Discovery and Graph Partitioning

During graph compilation, the session iterates through registered EPs and calls **`GetCapability`** for each. The EP inspects the `GraphViewer`, checks operator support through `IKernelLookup`, and returns a vector of `ComputeCapability` objects. This process determines which nodes execute on which hardware.

### Kernel Lookup and Instantiation

When executing a partitioned graph, the session requests the kernel registry via **`GetKernelRegistry`**. The registry returns a **`KernelCreateInfo`** factory that constructs the concrete `OpKernel` instance. For EPs that rely on external libraries rather than custom kernels, this method returns `nullptr`.

### Memory Allocation and Data Transfer

If the EP overrides **`CreatePreferredAllocators`** (as seen in `cpu_execution_provider.cc` lines 61-66), the session initializes these allocators during startup for tensor creation. For devices requiring host-device copies, the EP implements **`GetDataTransfer`** to return a concrete `IDataTransfer` object; the default implementation returns `nullptr`, indicating no special copy semantics are required.

## Implementing a Custom Execution Provider

To add support for a custom accelerator, implement the three essential overrides:

```cpp
#include "core/framework/execution_provider.h"

class MyCustomEP : public onnxruntime::IExecutionProvider {
 public:
  MyCustomEP() : IExecutionProvider("MyCustomEP") {}

  std::vector<std::unique_ptr<onnxruntime::ComputeCapability>>
  GetCapability(const onnxruntime::GraphViewer& graph_viewer,
                const IKernelLookup& kernel_lookup,
                const GraphOptimizerRegistry&,
                IResourceAccountant* = nullptr) const override {
    std::vector<std::unique_ptr<onnxruntime::ComputeCapability>> caps;
    for (auto node_idx : graph_viewer.GetNodesInTopologicalOrder()) {
      const auto& node = *graph_viewer.GetNode(node_idx);
      if (node.OpType() == "Conv") {
        auto subgraph = std::make_unique<onnxruntime::IndexedSubGraph>();
        subgraph->nodes.push_back(node.Index());
        caps.emplace_back(std::make_unique<onnxruntime::ComputeCapability>(std::move(subgraph)));
      }
    }
    return caps;
  }

  std::shared_ptr<onnxruntime::KernelRegistry> GetKernelRegistry() const override {
    static std::shared_ptr<onnxruntime::KernelRegistry> registry = 
        std::make_shared<onnxruntime::KernelRegistry>();
    // Register kernel factories here
    return registry;
  }
};

```

This implementation claims all `Conv` nodes and provides a registry for their kernels, integrating seamlessly with the microsoft/onnxruntime partitioning engine.

## Registering and Using Execution Providers

Register the custom EP with the session options before loading the model:

```cpp
Ort::Env env{ORT_LOGGING_LEVEL_WARNING, "my_ep_example"};
Ort::SessionOptions opts;
opts.AppendExecutionProvider(std::make_shared<MyCustomEP>());
Ort::Session session{env, "model.onnx", opts};

```

In Python, query available providers and their assignment:

```python
import onnxruntime as ort

sess = ort.InferenceSession("model.onnx")
providers = sess.get_providers()
print("Available EPs:", providers)

```

The Python `InferenceSession` forwards these calls to the same C++ machinery that interacts with `IExecutionProvider` instances.

## Summary

- **IExecutionProvider** is the abstract base class in [`execution_provider.h`](https://github.com/microsoft/onnxruntime/blob/main/execution_provider.h) that defines the contract for hardware backends through methods like `GetCapability`, `GetKernelRegistry`, and `CreatePreferredAllocators`.
- The **ExecutionProviders** container manages EP lifecycle and registration within the session, providing lookup by provider name.
- **Capability discovery** determines graph partitioning, while **kernel registries** supply the factories that instantiate device-specific `OpKernel` objects.
- EPs can implement **custom allocators** and **data transfer** utilities to optimize memory management and tensor movement for their specific hardware.
- New hardware accelerators integrate by deriving from `IExecutionProvider` and registering with `SessionOptions`, requiring no modifications to the core microsoft/onnxruntime framework.

## Frequently Asked Questions

### What is an Execution Provider in ONNX Runtime?

An Execution Provider is a hardware abstraction plug-in that implements the `IExecutionProvider` interface to supply device-specific kernels, memory allocators, and data-transfer capabilities. It allows ONNX Runtime to execute models on diverse hardware targets—from standard CPUs to specialized AI accelerators—through a unified interface defined in [`include/onnxruntime/core/framework/execution_provider.h`](https://github.com/microsoft/onnxruntime/blob/main/include/onnxruntime/core/framework/execution_provider.h).

### How does ONNX Runtime decide which EP executes a node?

During session initialization, the framework calls `IExecutionProvider::GetCapability` for each registered EP. This method returns `ComputeCapability` objects describing which sub-graphs the EP can execute. The session partitions the graph based on these capabilities, assigning nodes to the first EP in the priority list that claims support for that operator or fused pattern.

### What methods must a custom Execution Provider implement?

A minimal custom EP must implement three methods: the constructor supplying the provider type string, `GetCapability` to declare supported graph partitions, and `GetKernelRegistry` to return a registry containing kernel factories for those operations. Optional overrides include `CreatePreferredAllocators` for custom memory management and `GetDataTransfer` for optimized host-device copies.

### How do I register a custom EP in ONNX Runtime?

Register the EP by creating an `Ort::SessionOptions` instance and calling `AppendExecutionProvider` with a shared pointer to your `IExecutionProvider` implementation. In C++, this looks like `opts.AppendExecutionProvider(std::make_shared<MyCustomEP>())`. The session will then include your EP in the capability query and partitioning phase during model loading.