Performance Considerations for dotnet/skills: A Deep Dive into the Skill Validator

The dotnet/skills repository optimizes large-scale LLM evaluations through configurable parallelism, baseline reuse, and optional validation passes that balance statistical rigor against compute cost.

The dotnet/skills repository powers a sophisticated skill-validation framework that runs large-scale LLM-driven evaluations across multiple .NET upgrade scenarios. Understanding the performance considerations for dotnet/skills is essential for teams running continuous integration pipelines or cost-sensitive cloud evaluations. The validator exposes granular controls over concurrency, statistical sampling, and optional quality checks that directly impact runtime duration, memory consumption, and API token expenditure.

Configurable Parallelism and Concurrency Control

The validator implements a three-tier parallelism model that governs how skills, scenarios, and individual runs execute concurrently. Tuning these parameters correctly prevents resource saturation while maximizing throughput.

Tuning Parallel Skills, Scenarios, and Runs

In eng/skill-validator/src/Evaluate/EvaluateCommand.cs (lines 13–24), the CLI exposes three critical options:

• --parallel-skills: Controls how many skills evaluate simultaneously
• --parallel-scenarios: Governs concurrent scenario execution within a skill
• --parallel-runs: Limits simultaneous LLM calls for statistical sampling

The defaults (ParallelSkills = 1, ParallelScenarios = 1, ParallelRuns = 1) prioritize safety for single-core environments. Increasing these values linearly scales throughput but requires monitoring for CPU saturation, memory pressure, and external API rate limits. The effective limits are applied at lines 404, 598, and 683 using a dedicated concurrency primitive.

The ConcurrencyLimiter Implementation

The ConcurrencyLimiter class in eng/skill-validator/src/Shared/ConcurrencyLimiter.cs (lines 7–10) wraps SemaphoreSlim to enforce hard caps on async operations. This guarantees the validator never spawns more tasks than configured, preventing uncontrolled thread-pool growth and out-of-memory crashes when hundreds of LLM calls are in flight.

Statistical Rigor vs. Execution Cost

Balancing confidence intervals against compute budgets requires careful configuration of repetition counts and baseline handling.

Balancing Run Counts (--runs)

The --runs parameter (parsed at line 21 in EvaluateCommand.cs) determines how many times each scenario executes to compute statistically significant averages. The default value of 5 provides a reasonable confidence interval, but the code warns when fewer runs are used (lines 30–31). Each additional run proportionally increases total compute time and LLM token consumption, making this the primary cost driver for high-volume evaluations.

Reusing Baselines for Faster Iteration

The --baseline-out and --baseline-from options enable persistence of the "no-skill" baseline across multiple evaluations. In EvaluateCommand.cs (lines 44–77), the Run method handles baseline loading and creation. Reusing a pre-computed baseline eliminates the baseline arm for subsequent skill evaluations, reducing total runtime by approximately 33% for typical three-arm A/B/C tests.

Optional Validation Passes and Their Overhead

Certain validation features provide quality assurance at significant computational cost.

Overfitting Analysis (--no-overfitting-check)

When enabled, the validator runs an extra LLM-based pass to detect whether a skill is memorizing test data rather than generalizing. This check, controlled by the flag defined at line 30 and consulted at line 113, can dominate runtime and API costs because it invokes the model on every scenario again. Disabling this check with --no-overfitting-check reduces cost significantly when raw performance metrics are the only priority.

Plugin-Specific Performance Impacts

Individual plugins contain domain-specific performance guidance in their SKILL.md documentation. While the validator does not enforce these constraints, the documentation warns developers about regressions caused by default changes.

EF Core Collection Translation

The migrate-dotnet9-to-dotnet10 plugin documents parameterized collection performance impacts in plugins/dotnet-upgrade/skills/migrate-dotnet9-to-dotnet10/SKILL.md (line 180). Changes to EF Core's collection translation behavior can introduce unexpected query degradation that affects overall evaluation speed.

Blazor Virtualize Overscan

The build-perf-diagnostics plugin highlights Blazor Virtualize component overscan default changes in plugins/dotnet-msbuild/skills/build-perf-diagnostics/SKILL.md (line 159). Modifying overscan values directly impacts UI rendering performance during skill execution.

MSBuild Evaluation Optimizations

The eval-performance plugin provides guidance on MSBuild evaluation performance regarding glob walking and import chains in plugins/dotnet-msbuild/skills/eval-performance/SKILL.md (lines 2–23). Complex evaluation graphs can bottleneck the validator when processing large solution files.

Resource Management and Error Handling

Efficient resource utilization extends beyond concurrency limits to include throttling strategies and fail-fast mechanisms.

Throttling LLM Agents

Agents interacting with external LLM providers respect the --parallel-runs limit calculated at line 686 in EvaluateCommand.cs. This prevents hitting provider rate limits and maintains evaluation stability on modest hardware configurations.

Early Exit Strategies

The validator implements aggressive early-exit logic throughout the Run method (lines 44–47, 51–55). Configuration errors such as missing models or conflicting flags trigger immediate termination, preventing wasted compute budget on invalid evaluation pipelines.

Practical Optimization Examples

Running with Tuned Parallelism

skill-validator evaluate \
  --paths plugins/dotnet-upgrade \
  --tests-dir tests \
  --parallel-skills 4 \
  --parallel-scenarios 8 \
  --parallel-runs 6 \
  --runs 5 \
  --baseline-out baseline.json \
  --reporter console json markdown

The options correspond directly to declarations in EvaluateCommand.cs (lines 22–24).

Reusing a Pre-computed Baseline

# First run – generate baseline

skill-validator evaluate \
  --paths plugins/dotnet-upgrade \
  --tests-dir tests \
  --baseline-out baseline.json

# Subsequent runs – reuse baseline

skill-validator evaluate \
  --paths plugins/dotnet-upgrade \
  --tests-dir tests \
  --baseline-from baseline.json \
  --parallel-skills 3

Baseline handling occurs in Run (lines 44–77) where the BaselineStore is loaded or created.

Disabling Overfitting Checks for Speed

skill-validator evaluate \
  --paths plugins/dotnet-upgrade \
  --tests-dir tests \
  --no-overfitting-check

The flag's effect is reflected in the config assembly (line 113) where OverfittingCheck is set to the inverse of the option.

Summary

• Set parallelism to match logical CPU cores while monitoring API rate limits; defaults are conservative for single-core machines but slow for production workloads.
• Reuse baseline files (--baseline-out followed by --baseline-from) to eliminate redundant control runs and reduce total evaluation time by up to 33%.
• Maintain --runs at 5 or higher for statistically meaningful results, accepting the linear cost increase for improved confidence intervals.
• Disable overfitting analysis (--no-overfitting-check) when conducting pure performance benchmarks without quality gates.
• Consult plugin-specific documentation in each SKILL.md file to avoid regressions from framework defaults in EF Core, Blazor, and MSBuild.
• Leverage the microbenchmarking skill in plugins/dotnet-diag/skills/microbenchmarking/SKILL.md to isolate hot paths before running full validator suites.

Frequently Asked Questions

How does the ConcurrencyLimiter prevent resource exhaustion?

The ConcurrencyLimiter class in eng/skill-validator/src/Shared/ConcurrencyLimiter.cs wraps SemaphoreSlim to guarantee a minimum of one concurrent operation while enforcing user-specified maximums. This prevents the validator from spawning unlimited async tasks during large-scale LLM evaluations, eliminating thread-pool starvation and out-of-memory crashes that occur when hundreds of API calls execute simultaneously.

What is the optimal --runs value for production evaluations?

The default value of 5 strikes a balance between statistical confidence and computational cost, as implemented in eng/skill-validator/src/Evaluate/EvaluateCommand.cs (lines 21, 30–31). Values below 5 trigger warnings due to widened confidence intervals, while values above 10 provide diminishing statistical returns despite linear cost increases in LLM token consumption.

When should I disable overfitting checks?

Disable overfitting analysis using --no-overfitting-check when running pure performance regression tests where you prioritize speed and cost over quality validation. This skips the extra LLM pass defined at line 113 of EvaluateCommand.cs, significantly reducing runtime when you are confident in the skill's generalization capabilities or are iterating rapidly on benchmark configurations.

How much time does baseline reuse actually save?

Reusing baselines via --baseline-from reduces evaluation time by approximately 33% for standard three-arm tests (baseline plus two variants), according to the implementation in EvaluateCommand.cs (lines 68–70). This optimization eliminates the need to re-run the control scenario for every skill evaluation, particularly beneficial when iterating on multiple candidate solutions against a fixed test suite.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}