How to Configure BenchmarkDotNet Jobs for Cross-.NET Version Performance Comparison

BenchmarkDotNet treats each Job as a complete execution environment, allowing you to configure multiple runtimes in a single benchmark class to compare .NET 8.0 versus .NET 9.0 performance within one process.

The dotnet/skills repository provides authoritative guidance on microbenchmarking strategies, specifically detailing how to compare code performance across different .NET versions using multiple Jobs. By leveraging the Job configuration system documented in plugins/dotnet-diag/skills/microbenchmarking/SKILL.md, you can execute the same benchmark code under different runtimes in a single process, eliminating the environmental variance that occurs when running benchmarks separately.

Understanding BenchmarkDotNet Jobs

A Job in BenchmarkDotNet represents a complete set of execution parameters that define how a benchmark runs. According to the microbenchmarking skill documentation, a Job encapsulates the target runtime, launch count, iteration count, GC mode, and environment variables. When you configure BenchmarkDotNet jobs for cross-.NET version performance comparison, you are essentially defining separate execution environments that target different CoreRuntime versions within the same benchmark session.

Project Setup and Package Installation

Before configuring multi-runtime jobs, create a console application and install the BenchmarkDotNet package. The documentation in plugins/dotnet-diag/skills/microbenchmarking/references/project-setup-and-running.md recommends adding the package without specifying a version, allowing NuGet to resolve the newest compatible version:

dotnet new console -n CrossRuntimeBenchmarks
cd CrossRuntimeBenchmarks
dotnet add package BenchmarkDotNet

This setup provides the BenchmarkRunner and configuration APIs needed to define custom jobs targeting specific .NET versions.

Configuring Jobs for Multiple .NET Runtimes

To compare performance across .NET versions, define separate jobs using Job.Default.WithRuntime() and specify the desired CoreRuntime. The reference documentation in plugins/dotnet-diag/skills/microbenchmarking/references/comparison-strategies.md explicitly recommends this approach for controlling environmental variance.

Targeting Specific Runtimes

Use CoreRuntime.Core80 for .NET 8.0 and CoreRuntime.Core90 for .NET 9.0. Each job configuration creates an isolated execution environment:

using BenchmarkDotNet.Jobs;
using BenchmarkDotNet.Environments;

// Configure job for .NET 8.0
var net8Job = Job.Default
    .WithRuntime(CoreRuntime.Core80)
    .WithId("net8");

// Configure job for .NET 9.0
var net9Job = Job.Default
    .WithRuntime(CoreRuntime.Core90)
    .WithId("net9");

Setting a Baseline for Comparison

When configuring multiple jobs, mark one as the baseline using AsBaseline(). This instructs BenchmarkDotNet to calculate performance ratios relative to the reference job, making it easy to identify regressions or improvements:

var baselineJob = Job.Default
    .WithRuntime(CoreRuntime.Core80)
    .AsBaseline()
    .WithId("net8");

The comparison strategies documentation notes that the baseline job serves as the reference point, with BenchmarkDotNet generating a Ratio column in the results table showing how other jobs perform relative to it.

Creating a Custom Configuration Class

Encapsulate your job definitions in a class that inherits from ManualConfig. This approach, as demonstrated in the dotnet/skills examples, keeps your benchmark classes clean while centralizing runtime configuration:

using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Jobs;

public class CrossRuntimeConfig : ManualConfig
{
    public CrossRuntimeConfig()
    {
        AddJob(Job.Default
            .WithRuntime(CoreRuntime.Core80)
            .AsBaseline()
            .WithId("net8"));
            
        AddJob(Job.Default
            .WithRuntime(CoreRuntime.Core90)
            .WithId("net9"));
    }
}

The WithId() method provides human-readable identifiers that appear in result tables and artifact folder names, making it easy to distinguish between runtimes when analyzing outputs.

Complete Cross-Runtime Benchmark Example

Here is a complete implementation that combines job configuration with a sample benchmark. This example uses MemoryDiagnoser to capture allocation metrics, which is often critical when comparing runtime versions:

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Jobs;
using BenchmarkDotNet.Environments;
using BenchmarkDotNet.Running;

[MemoryDiagnoser]
public class StringConcatBench
{
    private readonly string[] data = Enumerable.Range(0, 1000)
                                               .Select(i => i.ToString())
                                               .ToArray();

    [Benchmark]
    public string ConcatLoop() => string.Concat(data);

    [Benchmark]
    public string Join() => string.Join(string.Empty, data);
}

public class CrossRuntimeConfig : ManualConfig
{
    public CrossRuntimeConfig()
    {
        AddJob(Job.Default
            .WithRuntime(CoreRuntime.Core80)
            .AsBaseline()
            .WithId("net8"));
            
        AddJob(Job.Default
            .WithRuntime(CoreRuntime.Core90)
            .WithId("net9"));
    }
}

public class Program
{
    public static void Main(string[] args) =>
        BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly)
                         .RunAll(new CrossRuntimeConfig());
}

This configuration runs both benchmarks under .NET 8.0 (as the baseline) and .NET 9.0, generating comparison tables that include the ratio of execution time and memory allocations between versions.

Running Benchmarks and Preserving Results

When executing cross-runtime comparisons, use the --noOverwrite flag to prevent BenchmarkDotNet from overwriting previous results. As documented in plugins/dotnet-diag/skills/microbenchmarking/references/project-setup-and-running.md, this flag preserves historical benchmark data, enabling longitudinal analysis across multiple test runs:

dotnet run --configuration Release -- --noOverwrite

Exporting Results for Analysis

BenchmarkDotNet automatically generates Markdown and CSV files in the BenchmarkDotNet.Artifacts/ directory. For CI/CD pipelines or custom reporting, you can specify additional exporters as detailed in plugins/dotnet-diag/skills/microbenchmarking/references/diagnosers-and-exporters.md:

dotnet run --configuration Release -- --exporters html --exporters json --noOverwrite

Controlling Environmental Variance

Running multiple jobs in a single process is the recommended approach for cross-.NET version comparison because it controls for machine-level variance. The documentation in plugins/dotnet-diag/skills/microbenchmarking/references/comparison-strategies.md states that this method is "generally preferable to separate runs because it controls for environmental variance."

When you configure multiple jobs in one benchmark class, BenchmarkDotNet ensures that CPU characteristics, OS scheduler behavior, and background processes affect both runtime measurements equally. This isolation provides statistical confidence that performance differences result from runtime changes rather than external system noise.

Summary

• Jobs define execution environments: Each Job configures a specific runtime, GC mode, and iteration strategy, making it the primary mechanism for cross-runtime testing.
• Use WithRuntime() and CoreRuntime: Target specific .NET versions like CoreRuntime.Core80 or CoreRuntime.Core90 to isolate runtime-specific performance.
• Mark baselines with AsBaseline(): This generates ratio columns in result tables, simplifying the identification of performance regressions or improvements.
• Preserve results with --noOverwrite: Prevents overwriting previous benchmark artifacts, maintaining a history of cross-runtime comparisons.
• Single-process comparison reduces variance: Running multiple jobs in one benchmark execution controls for environmental factors that could skew results in separate runs.

Frequently Asked Questions

How do I add a third .NET version to my comparison?

Add another AddJob() call in your ManualConfig class with the desired runtime version. For example, to include .NET 6.0, add Job.Default.WithRuntime(CoreRuntime.Core60).WithId("net6") alongside your existing .NET 8.0 and .NET 9.0 jobs. Ensure you have the corresponding .NET SDK installed, as BenchmarkDotNet will attempt to locate it using the dotnet CLI.

Why is my baseline job showing a ratio of 1.00 while others show different values?

The AsBaseline() method marks that specific job as the reference point with a fixed ratio of 1.00. BenchmarkDotNet calculates the ratio for other jobs by dividing their execution time by the baseline execution time. Values less than 1.00 indicate faster performance than the baseline, while values greater than 1.00 indicate slower performance.

Can I configure different GC modes for each runtime?

Yes, you can chain additional configuration methods when defining jobs. According to plugins/dotnet-diag/skills/microbenchmarking/references/comparison-strategies.md, you can use mutators like .WithGcServer(true) or .WithGcConcurrent(true) on individual jobs without affecting other runtimes in the same benchmark. This allows you to test both workstation and server GC configurations across different .NET versions.

Where can I find the official BenchmarkDotNet documentation referenced in the dotnet/skills repository?

The dotnet/skills repository located at github.com/dotnet/skills contains the microbenchmarking skill under plugins/dotnet-diag/skills/microbenchmarking/. Key files include SKILL.md for Job definitions, references/project-setup-and-running.md for CLI flags, and references/comparison-strategies.md for baseline configuration and variance control techniques.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}