Customizing ONNX Runtime with session_options in Transformers.js: A Complete Guide

You can customize ONNX Runtime behavior in Transformers.js by passing a session_options object to from_pretrained(), which propagates directly to InferenceSession.create() to control execution providers, logging, memory optimization, and hardware acceleration.

Transformers.js provides direct pass-through access to ONNX Runtime's configuration layer through the session_options parameter. This allows developers to fine-tune inference performance, enable hardware acceleration, and manage memory usage when loading models from the Hugging Face Hub or local paths.

Understanding the session_options Architecture

The library implements a thin abstraction layer that ultimately creates an ONNX InferenceSession provided by onnxruntime-common. When you call from_pretrained(), your session_options object travels through four distinct stages before reaching the runtime.

Stage 1: Model Metadata Resolution

In src/utils/model-loader.js, the getModelDataFiles function resolves external data chunks or user-supplied session_options.externalData arrays. This handles large models that split weights into separate binary files.

Stage 2: Session Preparation

The getSession function in src/models/session.js (lines 82-112) merges your provided options with sensible defaults. It handles execution provider selection, free-dimension overrides, and log level configuration.

Stage 3: Backend Initialization

In src/backends/onnx.js, the createInferenceSession function (lines 62-78) ensures the WASM or WebGPU backend is loaded before calling InferenceSession.create(buffer_or_path, session_options).

Stage 4: Runtime Environment Exposure

The same file exposes env.backends.onnx, allowing runtime modification of WASM proxy settings and WebGPU power preferences after initialization.

How session_options Propagates Through the Codebase

User Entry Point

When loading a model, you provide session_options through the PretrainedModelOptions type defined in src/utils/hub.js:

// src/utils/hub.js – Type definition
// @property {import('onnxruntime-common').InferenceSession.SessionOptions} [session_options]
// (Optional) User-specified session options passed to the runtime.

Merging with Defaults

The library applies default optimizations while preserving your overrides:

// src/models/session.js
const session_options = { ...options.session_options };
session_options.executionProviders ??= executionProviders;
session_options.freeDimensionOverrides ??= free_dimension_overrides;

External Data Handling

For models requiring external weight files, session_options.externalData triggers special handling in src/utils/model-loader.js (lines 99-107), fetching binary chunks and injecting them into the session.

Direct Pass-Through

No validation or transformation occurs at the final step. In src/backends/onnx.js (lines 72-76), the options object passes directly to InferenceSession.create(), ensuring full access to ONNX Runtime capabilities.

Advanced Configuration Scenarios

The following table maps common optimization goals to their corresponding session_options keys:

Goal	session_options Key	Typical Value	Implementation Detail
Change ONNX log severity	logSeverityLevel	0 (VERBOSE) to 4 (ERROR)	Controls console output from the native runtime
Select execution provider	executionProviders	[{ name: 'webgpu' }] or [{ name: 'cpu' }]	Determines whether CPU, WASM, or GPU acceleration is used
Fix dynamic dimensions	freeDimensionOverrides	{ batch_size: 1 }	Optimizes memory allocation for known input shapes
Control output buffer location	preferredOutputLocation	'cpu'	Forces tensor outputs to reside in CPU memory
Load external weights	externalData	[{ path: 'weights.bin', data: 'weights.bin' }]	References binary files for models exceeding single-file limits
Enable WASM proxy	env.backends.onnx.wasm.proxy	true	Offloads inference to Web Workers in browser environments
WebGPU power preference	env.backends.onnx.webgpu.powerPreference	'low-power' or 'high-performance'	Hints to the browser's GPU selection logic

Practical Code Examples

Basic Custom Log Level and CPU Provider

Control verbosity and force CPU execution when loading models from the Hugging Face Hub:

import { AutoModelForCausalLM, pipeline } from '@xenova/transformers';

const model = await AutoModelForCausalLM.from_pretrained('gpt2', {
  session_options: {
    logSeverityLevel: 0,               // VERBOSE logging
    executionProviders: [{ name: 'cpu' }],
  },
});

const generate = pipeline('text-generation', model);
const out = await generate('Hello world', { max_new_tokens: 20 });
console.log(out);

Source of the session_options field: src/utils/hub.js lines 46-48.

WebGPU with Free-Dimension Override

Optimize segmentation models by fixing batch dimensions and enabling GPU acceleration:

import { AutoModelForImageSegmentation } from '@xenova/transformers';

const model = await AutoModelForImageSegmentation.from_pretrained(
  'facebook/detr-resnet-50-panoptic',
  {
    device: 'webgpu',                  // Forces WebGPU backend selection
    session_options: {
      freeDimensionOverrides: { batch_size: 1 },
      executionProviders: [{ name: 'webgpu' }],
    },
  },
);

The backend automatically sets webgpu.powerPreference to 'high-performance' – see src/backends/onnx.js lines 41-43.

Loading External Data Files

Handle large language models that split weights into external binary files:

import { AutoModelForCausalLM } from '@xenova/transformers';

const model = await AutoModelForCausalLM.from_pretrained(
  'bigscience/bloom',
  {
    session_options: {
      externalData: [
        // Path relative to the model folder or a remote URL
        { path: 'weights.bin', data: 'weights.bin' },
      ],
    },
  },
);

External-data handling: src/utils/model-loader.js lines 99-107.

Enabling WASM Proxy for UI Thread Offloading

Prevent browser freezing during heavy inference by running ONNX in a Web Worker:

import { env } from '@xenova/transformers';

// Turn the proxy on *after* the runtime has been imported.
env.backends.onnx.wasm.proxy = true;

// Load any model as usual; inference will now run in a Web Worker.
const { AutoModelForSequenceClassification } = await import('@xenova/transformers');
const model = await AutoModelForSequenceClassification.from_pretrained('distilbert-base-uncased-finetuned-sst-2-english');

Proxy flag location: src/backends/onnx.js lines 36-39.

Key Source Files Reference

File	Role	Link
src/backends/onnx.js	Core ONNX Runtime wrapper – creates sessions, loads WASM/WebGPU, exposes env.backends.onnx.	View
src/utils/hub.js	Defines the public PretrainedModelOptions type, including session_options.	View
src/utils/model-loader.js	Resolves external data, respects session_options.externalData.	View
src/models/session.js	Merges user-provided session_options with defaults (execution providers, free-dimension overrides, etc.).	View
src/models/modeling_utils.js	Higher-level helper that forwards session_options when constructing model classes.	View

Summary

• Direct pass-through: Transformers.js passes session_options directly to InferenceSession.create() without modification, giving you full access to ONNX Runtime capabilities.
• Default merging: The library automatically sets sensible defaults for executionProviders and freeDimensionOverrides in src/models/session.js, but your explicit values always take precedence.
• External data support: Large models requiring binary weight files can be loaded via session_options.externalData, handled in src/utils/model-loader.js.
• Runtime environment control: For browser-specific optimizations like WASM proxying or WebGPU power preferences, modify env.backends.onnx after import but before model loading.

Frequently Asked Questions

What is the default execution provider if I don't specify session_options?

If you omit session_options, Transformers.js automatically selects the best available backend based on your environment. In Node.js, it defaults to the cpu provider. In browsers, it attempts to use webgpu if available, falling back to WASM (cpu) otherwise. This logic is implemented in src/models/session.js where defaults are merged with user-provided options.

Can I use session_options to enable WebGPU in the browser?

Yes. While you can set device: 'webgpu' in the model constructor options, you should also explicitly set session_options.executionProviders to [{ name: 'webgpu' }] to ensure the ONNX Runtime uses the WebGPU backend. Additionally, you can control GPU power preferences via env.backends.onnx.webgpu.powerPreference set to 'high-performance' or 'low-power' before loading the model.

How do I handle large models with external data files?

For models that exceed ONNX's single-file size limits and split weights into external .bin files, use the session_options.externalData array. Each entry should specify the path (filename) and data (URL or local path) for the binary chunk. The src/utils/model-loader.js module automatically fetches these files and injects them into the session during model initialization.

What's the difference between session_options and env.backends.onnx?

session_options is a per-model configuration object passed during from_pretrained() that controls ONNX Runtime behavior for that specific inference session, such as execution providers, logging levels, and external data. In contrast, env.backends.onnx is a global runtime environment object that controls cross-cutting concerns like WASM proxy settings (env.backends.onnx.wasm.proxy) and WebGPU power preferences, which affect all subsequent model loads in the application lifecycle.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}