How BenchmarkDotNet Job Presets Configure Execution for Different Scenarios

BenchmarkDotNet job presets are preconfigured execution templates that bundle warmup counts, iteration cycles, launch strategies, and runtimes into reusable attributes, allowing developers to trade measurement speed against statistical confidence.

BenchmarkDotNet encapsulates benchmark execution environments in jobs, which define every parameter influencing measurement accuracy. According to the dotnet/skills repository reference documentation found in plugins/dotnet-diag/skills/microbenchmarking/references/bdn-internals-and-tuning.md, the library provides five distinct BenchmarkDotNet job presets designed to balance execution time with statistical reliability for scenarios ranging from quick smoke tests to rigorous long-running analysis.

The Five Built-In Job Presets

BenchmarkDotNet ships with five preset configurations that cover common performance testing needs. Each preset defines specific values for per-case time, warmup iterations, target iterations, launch count, and run strategy.

Dry Preset

The Dry preset performs minimal execution to validate that code compiles and runs without measuring steady-state performance. It executes in less than one second using zero warmup iterations, one target iteration, and a single launch count. This preset employs the ColdStart run strategy, which performs a single invocation per iteration without warmup or pilot phases.

Short Preset

The Short preset provides quick checks where limited warmup and measurement iterations suffice. It runs for approximately 5–8 seconds using three warmup iterations, three target iterations, and one launch count. This configuration uses the Throughput run strategy, making it ideal for rapid feedback during development cycles.

Default Preset

The Default preset serves as the general-purpose configuration when no explicit preset is specified. It executes for 15–25 seconds using adaptive warmup counts of 6–50 iterations and adaptive target iterations of 15–100, adjusting based on benchmark stability. With one launch count and the Throughput strategy, this preset balances thoroughness with reasonable execution time.

Medium Preset

The Medium preset targets benchmarks requiring longer warmup periods and additional repetitions to achieve stable results. Running for 33–52 seconds, it uses ten warmup iterations, fifteen target iterations, and two launch counts (spawning two separate processes per case). This preset also employs the Throughput strategy.

Long Preset

The Long preset accommodates very slow or highly variable workloads demanding extensive iteration and multiple processes for reliable statistics. It executes for three to twelve minutes using fifteen warmup iterations, one hundred target iterations, and three launch counts. Like Medium, it uses the Throughput strategy to ensure consistent measurement.

How Job Presets Work as Mutators

In BenchmarkDotNet, presets function as mutators that apply their settings to every job defined within a benchmark class. When you declare multiple jobs—such as comparing different runtimes—a preset's warmup count, iteration count, and other parameters apply to all of them simultaneously.

Command-line interface flags act as additional mutators. For example, invoking dotnet run -- --warmupCount 3 overrides the warmup value on every job in the configuration, regardless of the preset defined in source code. This architecture allows flexible tuning without code changes.

Implementing Job Presets in Code

You can apply BenchmarkDotNet job presets through three primary mechanisms: attributes, fluent API, or CLI flags.

Attribute-Based Configuration

Attach preset attributes directly to benchmark classes or methods for declarative configuration:

using BenchmarkDotNet.Attributes;

// Apply Short preset for quick validation
[ShortRunJob]
public class QuickBenchmark
{
    [Benchmark]
    public void DoWork()
    {
        // Implementation here
    }
}

Combine multiple jobs with mutators to test across runtimes while applying consistent settings:

using BenchmarkDotNet.Attributes;

[SimpleJob(RuntimeMoniker.Net80)]
[SimpleJob(RuntimeMoniker.Net90)]
[WarmupCount(3)]  // Mutator applied to both jobs above
public class RuntimeComparison
{
    [Benchmark]
    public void MethodA()
    {
        // Implementation here
    }
}

Fluent API Configuration

Create customized jobs by extending presets programmatically through the Job class:

using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Jobs;
using BenchmarkDotNet.Running;

// Start with Short preset and customize
var customJob = Job.Short
    .WithLaunchCount(2)          // Override: use 2 processes
    .WithIterationCount(10);     // Override: force 10 iterations

var config = ManualConfig.Create(DefaultConfig.Instance)
    .AddJob(customJob);

BenchmarkRunner.Run<QuickBenchmark>(config);

CLI Flag Overrides

Override or supplement presets without modifying source code using command-line arguments:

# Apply Short job preset via CLI

dotnet run -- -job Short

# Override warmup count for all jobs

dotnet run -- --warmupCount 5

Summary

• BenchmarkDotNet defines five job presets (Dry, Short, Default, Medium, Long) that configure warmup iterations, target iterations, launch counts, and run strategies.
• Dry uses the ColdStart strategy for compilation validation, while Short, Default, Medium, and Long use Throughput for actual performance measurement.
• Presets act as mutators, applying their settings to all jobs in a benchmark class, including those created for cross-runtime comparisons.
• You can apply presets via [ShortRunJob] attributes, the Job.Short fluent API, or CLI flags like --job Short.
• CLI arguments and manual job configurations can override preset defaults, providing flexibility for different testing environments.

Frequently Asked Questions

What is the difference between Dry and Short job presets in BenchmarkDotNet?

The Dry preset validates that benchmarks compile and execute without measuring performance, using zero warmup, one iteration, and the ColdStart strategy. The Short preset actually measures performance with three warmup and three target iterations using the Throughput strategy, completing in 5–8 seconds versus Dry's sub-second execution.

Can I combine multiple job presets in a single benchmark class?

Yes, you can combine multiple jobs and presets, but remember that presets function as mutators affecting all jobs. When you apply [ShortRunJob] to a class containing multiple [SimpleJob] attributes for different runtimes, the Short preset's settings apply to every runtime job simultaneously.

How do CLI flags interact with BenchmarkDotNet job presets?

CLI flags like --warmupCount 3 or --job Short act as mutators that override the values defined by presets in your source code. These flags apply globally to all jobs in the configuration, allowing you to tune execution parameters without recompiling your benchmark project.

When should I use the Long job preset instead of Medium?

Use the Long preset when benchmarking highly variable workloads or very slow operations where statistical noise requires 100 target iterations across three process launches (3–12 minutes total). The Medium preset (33–52 seconds, 15 iterations, 2 launches) suits stable workloads needing moderate repetition, while Long provides the statistical confidence necessary for publishing research or critical performance regressions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}