How to Configure BenchmarkDotNet for Side‑by‑Side Performance Comparisons in the dotnet/skills Repository

Use the microbenchmarking skill in dotnet/skills to generate benchmarks that employ [Benchmark(Baseline = true)] or .AsBaseline() on jobs, then run via BenchmarkSwitcher with --filter "*" and --runtimes flags to produce deterministic side‑by‑side results.

The dotnet/skills repository provides a specialized microbenchmarking skill that standardizes how agents generate and execute performance comparisons. This skill, located under plugins/dotnet-diag/skills/microbenchmarking/, encapsulates best practices for configuring BenchmarkDotNet to run side‑by‑side comparisons across multiple runtimes or implementations. By following the reference documentation found in the skill's markdown files, developers can ensure reproducible benchmarking workflows that avoid interactive prompts and preserve historical results.

Architecture of the Microbenchmarking Skill

The microbenchmarking skill is structured as a modular set of guidelines rather than a code library. According to plugins/dotnet-diag/skills/microbenchmarking/SKILL.md, the skill activates whenever a task requires BenchmarkDotNet execution. It delegates implementation details to four key reference documents:

• project-setup-and-running.md – Scaffold projects and invoke the runner
• comparison-strategies.md – Configure baselines and multiple jobs
• writing-benchmarks.md – Syntax and attribute usage
• diagnosers-and-exporters.md – Additional metrics and output formats

The skill mandates non‑interactive execution by requiring --filter "*" and --noOverwrite arguments, ensuring that automated agents can run benchmarks without hanging on console prompts.

Setting Up Side‑by‑Side Comparisons

To configure BenchmarkDotNet for side‑by‑side performance comparisons, you must establish a baseline and define comparative jobs. The workflow follows three distinct configuration layers as documented in comparison-strategies.md.

Marking the Baseline

Every comparison requires a reference point. You can designate a baseline using either declarative attributes or fluent configuration:

• Method-level baseline: Apply [Benchmark(Baseline = true)] to the reference implementation
• Job-level baseline: Call .AsBaseline() on a Job instance when configuring runtimes

Only one baseline is permitted per comparison group. The baseline method or job serves as the denominator for the Ratio column in the final results table.

Configuring Multiple Jobs

Side‑by‑side comparisons require at least two jobs targeting different variables—whether distinct runtimes (e.g., .NET 6.0 vs .NET 7.0), JIT compilers, or configuration flags. In a custom ManualConfig class, instantiate separate Job objects and add them to the configuration:

var net6 = Job.Default.WithRuntime(".NET 6.0").AsBaseline();
var net7 = Job.Default.WithRuntime(".NET 7.0");
AddJob(net6);
AddJob(net7);

Running with BenchmarkSwitcher

The skill recommends BenchmarkSwitcher over BenchmarkRunner because it respects CLI arguments required for automation. Essential flags include:

• --filter "*" – Executes all benchmarks without interactive selection
• --noOverwrite – Preserves previous runs in timestamped subdirectories under BenchmarkDotNet.Artifacts/
• --runtimes net6.0 net7.0 – Targets multiple runtimes in a single execution

As noted in project-setup-and-running.md, redirecting console output to a log file is recommended for CI/CD environments.

Complete Implementation Example

The following example demonstrates a full side‑by‑side comparison between two sorting algorithms across two .NET runtimes:

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Jobs;
using BenchmarkDotNet.Running;

public class SortBenchmark
{
    private int[] data;

    [GlobalSetup]
    public void Setup()
    {
        var rnd = new Random(42);
        data = Enumerable.Range(0, 10_000).Select(_ => rnd.Next()).ToArray();
    }

    [Benchmark(Baseline = true)]
    public int[] ArraySort() => data.OrderBy(x => x).ToArray();

    [Benchmark]
    public int[] LinqSort() => data.OrderBy(x => x).ToArray();
}

public class ComparisonConfig : ManualConfig
{
    public ComparisonConfig()
    {
        var net6 = Job.Default
            .WithRuntime(".NET 6.0")
            .AsBaseline();

        var net7 = Job.Default
            .WithRuntime(".NET 7.0");

        AddJob(net6);
        AddJob(net7);
    }
}

public class Program
{
    public static int Main(string[] args) =>
        BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly)
            .Run(args, new ComparisonConfig());
}

Execute the benchmark with:

dotnet build -c Release
dotnet run -c Release --filter "*" --noOverwrite --runtimes net6.0 net7.0 > benchmark.log

Interpreting Side‑by‑Side Results

BenchmarkDotNet generates a Markdown summary containing columns for Mean, Error, StdDev, and Ratio. The Ratio column displays relative performance compared to the baseline (e.g., 1.23× slower or 0.85× faster). Results are stored in BenchmarkDotNet.Artifacts/<timestamp>/, with each run isolated when using --noOverwrite.

Summary

• The dotnet/skills microbenchmarking skill under plugins/dotnet-diag/skills/microbenchmarking/ provides the authoritative workflow for BenchmarkDotNet configuration
• Use [Benchmark(Baseline = true)] for method baselines or .AsBaseline() on Job configurations for runtime baselines
• Always invoke benchmarks via BenchmarkSwitcher with --filter "*" to prevent interactive hangs
• Store historical results by adding --noOverwrite to preserve timestamped artifacts in BenchmarkDotNet.Artifacts/
• Target multiple runtimes simultaneously using --runtimes netX.X netY.Y flags

Frequently Asked Questions

What is the difference between BenchmarkRunner and BenchmarkSwitcher?

BenchmarkRunner executes benchmarks immediately with hardcoded configuration, while BenchmarkSwitcher parses command-line arguments to filter and configure runs dynamically. The dotnet/skills repository mandates BenchmarkSwitcher because it supports the --filter and --noOverwrite flags required for automated, side‑by‑side comparisons without user intervention.

How do I compare more than two implementations in a single run?

Add multiple [Benchmark] methods to your class and designate one as the baseline using [Benchmark(Baseline = true)]. BenchmarkDotNet will execute all methods and calculate ratios relative to the baseline. For runtime comparisons, add multiple Job instances to your ManualConfig, ensuring only one job calls .AsBaseline().

Why does the skill require the --noOverwrite flag?

The --noOverwrite flag prevents BenchmarkDotNet from deleting previous results, instead creating timestamped subdirectories under BenchmarkDotNet.Artifacts/. This preserves historical data for trend analysis and ensures that automated agents can safely run benchmarks multiple times without losing prior measurements.

Where are the benchmark artifacts stored?

By default, BenchmarkDotNet writes results to BenchmarkDotNet.Artifacts/ in the project root. When using --noOverwrite, each execution creates a new subdirectory named with a timestamp. The skill documentation in project-setup-and-running.md recommends redirecting console output to a log file in addition to these automatic artifacts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}