# Performance Considerations for dotnet/skills: A Deep Dive into the Skill Validator

> Explore dotnet/skills performance considerations. Optimize LLM evaluations with configurable parallelism, baseline reuse, and validation passes balancing rigor and cost.

- Repository: [.NET Platform/skills](https://github.com/dotnet/skills)
- Tags: deep-dive
- Published: 2026-07-08

---

**The dotnet/skills repository optimizes large-scale LLM evaluations through configurable parallelism, baseline reuse, and optional validation passes that balance statistical rigor against compute cost.**

The `dotnet/skills` repository powers a sophisticated skill-validation framework that runs large-scale LLM-driven evaluations across multiple .NET upgrade scenarios. Understanding the performance considerations for dotnet/skills is essential for teams running continuous integration pipelines or cost-sensitive cloud evaluations. The validator exposes granular controls over concurrency, statistical sampling, and optional quality checks that directly impact runtime duration, memory consumption, and API token expenditure.

## Configurable Parallelism and Concurrency Control

The validator implements a three-tier parallelism model that governs how skills, scenarios, and individual runs execute concurrently. Tuning these parameters correctly prevents resource saturation while maximizing throughput.

### Tuning Parallel Skills, Scenarios, and Runs

In [`eng/skill-validator/src/Evaluate/EvaluateCommand.cs`](https://github.com/dotnet/skills/blob/main/eng/skill-validator/src/Evaluate/EvaluateCommand.cs) (lines 13–24), the CLI exposes three critical options:

- **`--parallel-skills`**: Controls how many skills evaluate simultaneously
- **`--parallel-scenarios`**: Governs concurrent scenario execution within a skill
- **`--parallel-runs`**: Limits simultaneous LLM calls for statistical sampling

The defaults (`ParallelSkills = 1`, `ParallelScenarios = 1`, `ParallelRuns = 1`) prioritize safety for single-core environments. Increasing these values linearly scales throughput but requires monitoring for CPU saturation, memory pressure, and external API rate limits. The effective limits are applied at lines 404, 598, and 683 using a dedicated concurrency primitive.

### The ConcurrencyLimiter Implementation

The `ConcurrencyLimiter` class in [`eng/skill-validator/src/Shared/ConcurrencyLimiter.cs`](https://github.com/dotnet/skills/blob/main/eng/skill-validator/src/Shared/ConcurrencyLimiter.cs) (lines 7–10) wraps `SemaphoreSlim` to enforce hard caps on async operations. This guarantees the validator never spawns more tasks than configured, preventing uncontrolled thread-pool growth and out-of-memory crashes when hundreds of LLM calls are in flight.

## Statistical Rigor vs. Execution Cost

Balancing confidence intervals against compute budgets requires careful configuration of repetition counts and baseline handling.

### Balancing Run Counts (--runs)

The `--runs` parameter (parsed at line 21 in [`EvaluateCommand.cs`](https://github.com/dotnet/skills/blob/main/EvaluateCommand.cs)) determines how many times each scenario executes to compute statistically significant averages. The default value of 5 provides a reasonable confidence interval, but the code warns when fewer runs are used (lines 30–31). Each additional run proportionally increases total compute time and LLM token consumption, making this the primary cost driver for high-volume evaluations.

### Reusing Baselines for Faster Iteration

The `--baseline-out` and `--baseline-from` options enable persistence of the "no-skill" baseline across multiple evaluations. In [`EvaluateCommand.cs`](https://github.com/dotnet/skills/blob/main/EvaluateCommand.cs) (lines 44–77), the `Run` method handles baseline loading and creation. Reusing a pre-computed baseline eliminates the baseline arm for subsequent skill evaluations, reducing total runtime by approximately 33% for typical three-arm A/B/C tests.

## Optional Validation Passes and Their Overhead

Certain validation features provide quality assurance at significant computational cost.

### Overfitting Analysis (--no-overfitting-check)

When enabled, the validator runs an extra LLM-based pass to detect whether a skill is memorizing test data rather than generalizing. This check, controlled by the flag defined at line 30 and consulted at line 113, can dominate runtime and API costs because it invokes the model on every scenario again. Disabling this check with `--no-overfitting-check` reduces cost significantly when raw performance metrics are the only priority.

## Plugin-Specific Performance Impacts

Individual plugins contain domain-specific performance guidance in their [`SKILL.md`](https://github.com/dotnet/skills/blob/main/SKILL.md) documentation. While the validator does not enforce these constraints, the documentation warns developers about regressions caused by default changes.

### EF Core Collection Translation

The `migrate-dotnet9-to-dotnet10` plugin documents parameterized collection performance impacts in [`plugins/dotnet-upgrade/skills/migrate-dotnet9-to-dotnet10/SKILL.md`](https://github.com/dotnet/skills/blob/main/plugins/dotnet-upgrade/skills/migrate-dotnet9-to-dotnet10/SKILL.md) (line 180). Changes to EF Core's collection translation behavior can introduce unexpected query degradation that affects overall evaluation speed.

### Blazor Virtualize Overscan

The `build-perf-diagnostics` plugin highlights Blazor `Virtualize` component overscan default changes in [`plugins/dotnet-msbuild/skills/build-perf-diagnostics/SKILL.md`](https://github.com/dotnet/skills/blob/main/plugins/dotnet-msbuild/skills/build-perf-diagnostics/SKILL.md) (line 159). Modifying overscan values directly impacts UI rendering performance during skill execution.

### MSBuild Evaluation Optimizations

The `eval-performance` plugin provides guidance on MSBuild evaluation performance regarding glob walking and import chains in [`plugins/dotnet-msbuild/skills/eval-performance/SKILL.md`](https://github.com/dotnet/skills/blob/main/plugins/dotnet-msbuild/skills/eval-performance/SKILL.md) (lines 2–23). Complex evaluation graphs can bottleneck the validator when processing large solution files.

## Resource Management and Error Handling

Efficient resource utilization extends beyond concurrency limits to include throttling strategies and fail-fast mechanisms.

### Throttling LLM Agents

Agents interacting with external LLM providers respect the `--parallel-runs` limit calculated at line 686 in [`EvaluateCommand.cs`](https://github.com/dotnet/skills/blob/main/EvaluateCommand.cs). This prevents hitting provider rate limits and maintains evaluation stability on modest hardware configurations.

### Early Exit Strategies

The validator implements aggressive early-exit logic throughout the `Run` method (lines 44–47, 51–55). Configuration errors such as missing models or conflicting flags trigger immediate termination, preventing wasted compute budget on invalid evaluation pipelines.

## Practical Optimization Examples

### Running with Tuned Parallelism

```bash
skill-validator evaluate \
  --paths plugins/dotnet-upgrade \
  --tests-dir tests \
  --parallel-skills 4 \
  --parallel-scenarios 8 \
  --parallel-runs 6 \
  --runs 5 \
  --baseline-out baseline.json \
  --reporter console json markdown

```

*The options correspond directly to declarations in [`EvaluateCommand.cs`](https://github.com/dotnet/skills/blob/main/EvaluateCommand.cs) (lines 22–24).*

### Reusing a Pre-computed Baseline

```bash

# First run – generate baseline

skill-validator evaluate \
  --paths plugins/dotnet-upgrade \
  --tests-dir tests \
  --baseline-out baseline.json

# Subsequent runs – reuse baseline

skill-validator evaluate \
  --paths plugins/dotnet-upgrade \
  --tests-dir tests \
  --baseline-from baseline.json \
  --parallel-skills 3

```

*Baseline handling occurs in `Run` (lines 44–77) where the `BaselineStore` is loaded or created.*

### Disabling Overfitting Checks for Speed

```bash
skill-validator evaluate \
  --paths plugins/dotnet-upgrade \
  --tests-dir tests \
  --no-overfitting-check

```

*The flag's effect is reflected in the config assembly (line 113) where `OverfittingCheck` is set to the inverse of the option.*

## Summary

- **Set parallelism to match logical CPU cores** while monitoring API rate limits; defaults are conservative for single-core machines but slow for production workloads.
- **Reuse baseline files** (`--baseline-out` followed by `--baseline-from`) to eliminate redundant control runs and reduce total evaluation time by up to 33%.
- **Maintain `--runs` at 5 or higher** for statistically meaningful results, accepting the linear cost increase for improved confidence intervals.
- **Disable overfitting analysis** (`--no-overfitting-check`) when conducting pure performance benchmarks without quality gates.
- **Consult plugin-specific documentation** in each [`SKILL.md`](https://github.com/dotnet/skills/blob/main/SKILL.md) file to avoid regressions from framework defaults in EF Core, Blazor, and MSBuild.
- **Leverage the microbenchmarking skill** in [`plugins/dotnet-diag/skills/microbenchmarking/SKILL.md`](https://github.com/dotnet/skills/blob/main/plugins/dotnet-diag/skills/microbenchmarking/SKILL.md) to isolate hot paths before running full validator suites.

## Frequently Asked Questions

### How does the ConcurrencyLimiter prevent resource exhaustion?

The `ConcurrencyLimiter` class in [`eng/skill-validator/src/Shared/ConcurrencyLimiter.cs`](https://github.com/dotnet/skills/blob/main/eng/skill-validator/src/Shared/ConcurrencyLimiter.cs) wraps `SemaphoreSlim` to guarantee a minimum of one concurrent operation while enforcing user-specified maximums. This prevents the validator from spawning unlimited async tasks during large-scale LLM evaluations, eliminating thread-pool starvation and out-of-memory crashes that occur when hundreds of API calls execute simultaneously.

### What is the optimal --runs value for production evaluations?

The default value of 5 strikes a balance between statistical confidence and computational cost, as implemented in [`eng/skill-validator/src/Evaluate/EvaluateCommand.cs`](https://github.com/dotnet/skills/blob/main/eng/skill-validator/src/Evaluate/EvaluateCommand.cs) (lines 21, 30–31). Values below 5 trigger warnings due to widened confidence intervals, while values above 10 provide diminishing statistical returns despite linear cost increases in LLM token consumption.

### When should I disable overfitting checks?

Disable overfitting analysis using `--no-overfitting-check` when running pure performance regression tests where you prioritize speed and cost over quality validation. This skips the extra LLM pass defined at line 113 of [`EvaluateCommand.cs`](https://github.com/dotnet/skills/blob/main/EvaluateCommand.cs), significantly reducing runtime when you are confident in the skill's generalization capabilities or are iterating rapidly on benchmark configurations.

### How much time does baseline reuse actually save?

Reusing baselines via `--baseline-from` reduces evaluation time by approximately 33% for standard three-arm tests (baseline plus two variants), according to the implementation in [`EvaluateCommand.cs`](https://github.com/dotnet/skills/blob/main/EvaluateCommand.cs) (lines 68–70). This optimization eliminates the need to re-run the control scenario for every skill evaluation, particularly beneficial when iterating on multiple candidate solutions against a fixed test suite.