How ONNX Runtime's Execution Provider Interface Works: A Deep Dive into Hardware Abstraction

ONNX Runtime's Execution Provider (EP) interface is an abstract plug-in architecture that enables hardware-specific backends to supply kernels, memory allocators, and data-transfer utilities, allowing the framework to offload computation to CPUs, GPUs, and custom accelerators without modifying the core runtime.

The microsoft/onnxruntime repository implements this sophisticated abstraction layer to decouple the graph execution engine from hardware-specific details. By defining a standardized contract in the core framework headers, the EP interface allows diverse backends to integrate seamlessly while the session orchestrates partitioning, memory management, and kernel dispatch.

Core Abstractions of the Execution Provider Interface

The EP architecture revolves around several key components that define the contract between the framework and hardware backends.

IExecutionProvider: The Abstract Base Class

Every hardware backend derives from IExecutionProvider, defined in include/onnxruntime/core/framework/execution_provider.h (lines 74-155). This abstract base class declares the essential virtual methods that concrete implementations must override:

• GetCapability – Inspects the graph and returns a vector of ComputeCapability objects describing which sub-graphs the EP can execute.
• GetKernelRegistry – Returns a std::shared_ptr<KernelRegistry> containing the kernel factories (KernelCreateInfo) for supported operators.
• CreatePreferredAllocators – Supplies device-specific memory allocators for tensor storage.
• GetDataTransfer – Returns an IDataTransfer instance for moving tensors between host and device memory.

The base class also defines lifecycle callbacks including OnRunStart, OnRunEnd, and Sync, which the session invokes during inference.

ExecutionProviders Container

The ExecutionProviders class, located in onnxruntime/core/framework/execution_providers.h (lines 24-84), manages the collection of EP instances attached to an InferenceSession. It provides fast lookup by provider name and stores provider-specific options through the Add method, which registers each EP shared pointer with the session.

KernelRegistry and ComputeCapability

When the session partitions the graph, it queries each EP's capability through ComputeCapability objects. These structures describe executable sub-graphs, whether single nodes or fused groups. The EP populates these by implementing GetCapability and inspecting the GraphViewer to verify supported operations via the supplied IKernelLookup interface.

The KernelRegistry holds the mapping between ONNX operator types and their concrete OpKernel implementations. When a node is assigned to an EP, the session calls GetKernelRegistry to retrieve the factory that instantiates the device-specific kernel.

Memory and Data Transfer Interfaces

EPs expose hardware resources through OrtDevice and OrtEpDevice descriptors, which include device IDs, types, and auxiliary devices for multi-GPU configurations. The CreatePreferredAllocators method allows EPs to override default memory allocation strategies, while GetDataTransfer enables custom copy routines between host and device buffers.

Execution Provider Lifecycle in ONNX Runtime

The EP passes through distinct phases during session initialization and runtime execution.

Construction and Registration

Concrete EP implementations, such as CPUExecutionProvider in onnxruntime/core/providers/cpu/cpu_execution_provider.cc, derive from IExecutionProvider and supply a type string (e.g., kCpuExecutionProvider) in their constructor:

CPUExecutionProvider(const CPUExecutionProviderInfo& info)
    : IExecutionProvider{onnxruntime::kCpuExecutionProvider}, info_{info} {}

The application registers EPs through ExecutionProviders::Add or via the C++ API:

Ort::SessionOptions session_options;
session_options.AppendExecutionProvider_CPU(Ort::AllocatorWithDefaultOptions());

Capability Discovery and Graph Partitioning

During graph compilation, the session iterates through registered EPs and calls GetCapability for each. The EP inspects the GraphViewer, checks operator support through IKernelLookup, and returns a vector of ComputeCapability objects. This process determines which nodes execute on which hardware.

Kernel Lookup and Instantiation

When executing a partitioned graph, the session requests the kernel registry via GetKernelRegistry. The registry returns a KernelCreateInfo factory that constructs the concrete OpKernel instance. For EPs that rely on external libraries rather than custom kernels, this method returns nullptr.

Memory Allocation and Data Transfer

If the EP overrides CreatePreferredAllocators (as seen in cpu_execution_provider.cc lines 61-66), the session initializes these allocators during startup for tensor creation. For devices requiring host-device copies, the EP implements GetDataTransfer to return a concrete IDataTransfer object; the default implementation returns nullptr, indicating no special copy semantics are required.

Implementing a Custom Execution Provider

To add support for a custom accelerator, implement the three essential overrides:

#include "core/framework/execution_provider.h"

class MyCustomEP : public onnxruntime::IExecutionProvider {
 public:
  MyCustomEP() : IExecutionProvider("MyCustomEP") {}

  std::vector<std::unique_ptr<onnxruntime::ComputeCapability>>
  GetCapability(const onnxruntime::GraphViewer& graph_viewer,
                const IKernelLookup& kernel_lookup,
                const GraphOptimizerRegistry&,
                IResourceAccountant* = nullptr) const override {
    std::vector<std::unique_ptr<onnxruntime::ComputeCapability>> caps;
    for (auto node_idx : graph_viewer.GetNodesInTopologicalOrder()) {
      const auto& node = *graph_viewer.GetNode(node_idx);
      if (node.OpType() == "Conv") {
        auto subgraph = std::make_unique<onnxruntime::IndexedSubGraph>();
        subgraph->nodes.push_back(node.Index());
        caps.emplace_back(std::make_unique<onnxruntime::ComputeCapability>(std::move(subgraph)));
      }
    }
    return caps;
  }

  std::shared_ptr<onnxruntime::KernelRegistry> GetKernelRegistry() const override {
    static std::shared_ptr<onnxruntime::KernelRegistry> registry = 
        std::make_shared<onnxruntime::KernelRegistry>();
    // Register kernel factories here
    return registry;
  }
};

This implementation claims all Conv nodes and provides a registry for their kernels, integrating seamlessly with the microsoft/onnxruntime partitioning engine.

Registering and Using Execution Providers

Register the custom EP with the session options before loading the model:

Ort::Env env{ORT_LOGGING_LEVEL_WARNING, "my_ep_example"};
Ort::SessionOptions opts;
opts.AppendExecutionProvider(std::make_shared<MyCustomEP>());
Ort::Session session{env, "model.onnx", opts};

In Python, query available providers and their assignment:

import onnxruntime as ort

sess = ort.InferenceSession("model.onnx")
providers = sess.get_providers()
print("Available EPs:", providers)

The Python InferenceSession forwards these calls to the same C++ machinery that interacts with IExecutionProvider instances.

Summary

• IExecutionProvider is the abstract base class in execution_provider.h that defines the contract for hardware backends through methods like GetCapability, GetKernelRegistry, and CreatePreferredAllocators.
• The ExecutionProviders container manages EP lifecycle and registration within the session, providing lookup by provider name.
• Capability discovery determines graph partitioning, while kernel registries supply the factories that instantiate device-specific OpKernel objects.
• EPs can implement custom allocators and data transfer utilities to optimize memory management and tensor movement for their specific hardware.
• New hardware accelerators integrate by deriving from IExecutionProvider and registering with SessionOptions, requiring no modifications to the core microsoft/onnxruntime framework.

Frequently Asked Questions

What is an Execution Provider in ONNX Runtime?

An Execution Provider is a hardware abstraction plug-in that implements the IExecutionProvider interface to supply device-specific kernels, memory allocators, and data-transfer capabilities. It allows ONNX Runtime to execute models on diverse hardware targets—from standard CPUs to specialized AI accelerators—through a unified interface defined in include/onnxruntime/core/framework/execution_provider.h.

How does ONNX Runtime decide which EP executes a node?

During session initialization, the framework calls IExecutionProvider::GetCapability for each registered EP. This method returns ComputeCapability objects describing which sub-graphs the EP can execute. The session partitions the graph based on these capabilities, assigning nodes to the first EP in the priority list that claims support for that operator or fused pattern.

What methods must a custom Execution Provider implement?

A minimal custom EP must implement three methods: the constructor supplying the provider type string, GetCapability to declare supported graph partitions, and GetKernelRegistry to return a registry containing kernel factories for those operations. Optional overrides include CreatePreferredAllocators for custom memory management and GetDataTransfer for optimized host-device copies.

How do I register a custom EP in ONNX Runtime?

Register the EP by creating an Ort::SessionOptions instance and calling AppendExecutionProvider with a shared pointer to your IExecutionProvider implementation. In C++, this looks like opts.AppendExecutionProvider(std::make_shared<MyCustomEP>()). The session will then include your EP in the capability query and partitioning phase during model loading.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}