Graph Optimizations in ONNX Runtime: The Complete Transformer Pipeline Explained

ONNX Runtime applies a hierarchical pipeline of rewrite rules—organized into Level 1 (basic) and Level 2 (extended) optimizations—during InferenceSession::Initialize, which can be controlled via SessionOptions::graph_optimization_level and fine-tuned by disabling specific transformers.

When you load an ONNX model into the microsoft/onnxruntime inference engine, the framework automatically rewrites the computational graph for maximum performance before execution begins. This transformation process, driven by the TransformerLevel setting in SessionOptions, executes a curated list of rewrite rules that eliminate redundant operations and fuse computation-heavy patterns. Understanding these graph optimizations is essential for debugging model behavior and minimizing inference latency.

How the Graph Transformer Pipeline Works

During InferenceSession::Initialize, ONNX Runtime constructs a graph transformer pipeline that converts the raw ONNX graph into an optimized executable form. The core logic resides in onnxruntime/core/optimizer/graph_transformer_utils.cc, specifically within the GenerateRewriteRules function.

The system organizes optimizations into four transformer levels defined in include/onnxruntime/core/optimizer/graph_transformer_level.h:

• Level 1: Basic optimizations safe for universal application
• Level 2: Extended optimizations that may slightly alter numerical precision
• Level 3-4: Reserved for future extensions (currently empty)

Each level instantiates a RuleBasedGraphTransformer (named "LevelX_RuleBasedTransformer") that registers specific rewrite rules with the GraphTransformerManager (onnxruntime/core/optimizer/graph_transformer_mgr.h). The manager executes these rules in ascending order via ApplyTransformers during session initialization.

Level 1 Optimizations (Basic)

Level 1 optimizations focus on eliminating no-op operations and performing straightforward algebraic fusions. The GenerateRewriteRules function registers these in the "Level1_RuleBasedTransformer":

• EliminateIdentity: Removes Identity nodes that pass tensors through unchanged
• EliminateSlice: Removes unnecessary Slice operations
• UnsqueezeElimination: Removes redundant Unsqueeze operations
• EliminateDropout: Strips Dropout nodes during inference
• ExpandElimination: Folds constant Expand operations into shapes
• CastElimination and optional CastChainElimination: Removes unnecessary type casting (enabled via SessionOptions::enable_cast_chain_elimination)
• PreShapeNodeElimination: Removes shape computation nodes when tensor shapes are known at graph construction time
• NoopElimination: Removes operations with no computational effect
• DivMulFusion: Combines consecutive Division and Multiplication operations
• FuseReluClip: Merges ReLU and Clip operations when bounds align
• GemmSumFusion: Fuses General Matrix Multiply (GEMM) with Add operations
• GemmTransposeFusion: Absorbs transpose operations into GEMM kernels
• NotWhereFusion: Optimizes boolean Not followed by Where operations
• ConvAddFusion, ConvMulFusion, ConvBNFusion: Fuses Convolution with element-wise operations and BatchNorm
• PadFusion: Merges padding operations into preceding convolution or pooling layers
• MatmulBNFusion: Fuses MatMul operations with BatchNorm
• LabelEncoderFusion: Optimizes categorical encoding patterns

Level 2 Optimizations (Extended)

Level 2 transformations apply more aggressive optimizations that may affect numerical precision. These rules are registered in the "Level2_RuleBasedTransformer":

• ClipQuantFusion: Integrates Clip operations into quantized nodes
• ReluQuantFusion: Merges ReLU activations into quantization operations
• GemmTransposeFusion: Additional GEMM transpose fusion patterns for specific hardware backends

Configuring Optimizations via SessionOptions

You can control the optimization pipeline through the Python API by adjusting graph_optimization_level and disabling specific rules:

import onnxruntime as ort

# Default: Maximum optimizations (Level 1 + Level 2)

sess = ort.InferenceSession("model.onnx")

# Basic optimizations only (Level 1)

opts = ort.SessionOptions()
opts.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_BASIC
sess_basic = ort.InferenceSession("model.onnx", sess_options=opts)

# Disable specific optimizer (e.g., ConvBNFusion)

opts = ort.SessionOptions()
opts.optimizers_to_disable = ["ConvBNFusion"]
sess_no_fusion = ort.InferenceSession("model.onnx", sess_options=opts)

# Disable entire transformer level (e.g., Level 2)

opts = ort.SessionOptions()
opts.transformers_to_disable = ["Level2_RuleBasedTransformer"]
sess_no_level2 = ort.InferenceSession("model.onnx", sess_options=opts)

The SessionOptions settings are parsed in graph_transformer_utils.cc when GenerateRewriteRules constructs the transformation list for the GraphTransformerManager.

Execution Provider Extensions

Beyond the core pipeline, individual Execution Providers (EPs) register provider-specific transformers through helper functions like RegisterGraphTransformers. For example, CUDA-specific optimizations are implemented in onnxruntime/core/providers/cuda/GraphTransformerHelpers.cc. These EP-specific rules execute alongside the standard Level 1 and Level 2 optimizations, targeting hardware-specific fusion opportunities.

Summary

• ONNX Runtime applies graph optimizations through a transformer pipeline during InferenceSession::Initialize in onnxruntime/core/session/inference_session.cc.
• The pipeline uses four transformer levels (currently utilizing Level 1 and Level 2), controlled by SessionOptions::graph_optimization_level.
• Level 1 includes 18+ rewrite rules for eliminating no-ops and basic fusions like ConvBNFusion and GemmSumFusion.
• Level 2 adds aggressive optimizations like ClipQuantFusion and ReluQuantFusion.
• Configuration happens via SessionOptions fields: optimizers_to_disable for specific rules and transformers_to_disable for entire levels.
• The architecture centers on GraphTransformerManager in onnxruntime/core/optimizer/graph_transformer_mgr.h and GenerateRewriteRules in onnxruntime/core/optimizer/graph_transformer_utils.cc.

Frequently Asked Questions

What is the default graph optimization level in ONNX Runtime?

By default, ONNX Runtime uses TransformerLevel::MaxLevel, which enables all available optimizations (currently Level 1 and Level 2). This corresponds to GraphOptimizationLevel.ORT_ENABLE_ALL in the Python API and is applied automatically when you create an InferenceSession without modifying SessionOptions.

How can I disable a specific fusion like ConvBNFusion?

You can disable individual rewrite rules by setting the optimizers_to_disable field in SessionOptions to a list containing the exact optimizer name. For example: opts.optimizers_to_disable = ["ConvBNFusion"]. This list is parsed in graph_transformer_utils.cc and excludes those rules from the RuleBasedGraphTransformer instances created for the session.

Where are the graph transformer levels defined in the source code?

The TransformerLevel enum is defined in include/onnxruntime/core/optimizer/graph_transformer_level.h, specifying Level1, Level2, Level3, Level4, and MaxLevel. The actual rules for each level are instantiated in onnxruntime/core/optimizer/graph_transformer_utils.cc within the GenerateRewriteRules function, which creates the RuleBasedGraphTransformer objects for each active level.

Do execution providers add their own graph optimizations?

Yes. Execution Providers register additional provider-specific transformers through functions like RegisterGraphTransformers in their respective directories (e.g., onnxruntime/core/providers/cuda/GraphTransformerHelpers.cc for CUDA). These EP-specific optimizations complement the core Level 1 and Level 2 rules managed by the GraphTransformerManager, targeting hardware-specific patterns that the generic optimizer cannot detect.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}