Skill-Validator Evaluation Framework Architecture: Inside the .NET Pipeline

The skill-validator evaluation framework architecture implements a modular command-line pipeline that discovers AI skills, executes three-arm LLM evaluations, and produces statistical quality reports via a self-contained .NET application.

The dotnet/skills repository hosts a sophisticated evaluation system designed to validate AI agents and skills through automated testing. This self-contained .NET console application orchestrates complex workflows including baseline comparisons, isolated testing, and plugin integration. Understanding the skill-validator evaluation framework architecture reveals how the system handles filesystem discovery, parallel LLM execution, statistical analysis, and multi-format reporting.

Architecture Layers

The framework organizes code into distinct logical layers under eng/skill-validator/src/, each with specific responsibilities and clear separation of concerns.

CLI Entry Point and Command Layer

Located in Program.cs, this layer initializes the System.CommandLine infrastructure to create a RootCommand with evaluate and check subcommands. The Evaluate/EvaluateCommand.cs and Check/CheckCommand.cs files parse command-line arguments and translate them into a ValidatorConfig object that drives the entire pipeline.

Discovery Layer

The Shared/SkillDiscovery.cs, Shared/PluginDiscovery.cs, and Shared/AgentDiscovery.cs files handle filesystem traversal to locate plugin.json, skill.yml, and *.agent.md files. This layer produces SkillInfo, AgentInfo, and plugin-root path objects consumed by the evaluation engine.

Evaluation Engine

The core orchestration lives in EvaluateCommand.Run, EvaluateTarget, EvaluateSkill, and EvaluateAgent. This engine implements the three-arm evaluation strategy (baseline, isolated, plugin) using AgentRunner for LLM execution, ConcurrencyLimiter for parallelism control, and RetryHelper for resilience. The Shared/Statistics.cs and Shared/Models.cs files define the data structures for metrics collection.

Judging and Comparison

Quality assessment occurs in Evaluate/Judge.cs, which calls the LLM judge model to generate rubric scores. The Comparator.cs handles pairwise comparisons when --judge-mode=pairwise is specified, while OverfittingJudge.cs detects judgment overfitting to evaluation prompts. These components produce ScenarioComparison objects that feed into the final SkillVerdict.

Baseline Caching

The BaselineStore.cs implements a caching mechanism for the baseline arm (no-skill/no-agent execution). This allows repeated runs to reuse baseline data via --baseline-from and --baseline-out parameters, significantly reducing evaluation time and API costs.

Reporting Layer

The Reporter.cs class formats results across multiple output formats including console, JSON, JUnit XML, and Markdown. It provides GenerateMarkdownSummary for human-readable reports and handles per-scenario markdown generation.

Data Flow Through the Pipeline

Execution begins at Program.cs with command parsing, flows through the discovery layer to build in-memory models, then enters the evaluation engine. The ValidatorConfig acts as the central configuration object passed through each stage. AgentRunner executes LLM calls while Judge and Comparator analyze outputs and compute improvement scores against the baseline. Finally, Reporter formats the list of SkillVerdict objects into the requested output formats.

Practical Usage Examples

Running a Full Evaluation

Invoke the evaluation pipeline from the command line using the evaluate subcommand:

skill-validator evaluate \
    --tests-dir tests \
    --results-dir .skill-validator-results \
    --reporter console json markdown \
    --model claude-opus-4.6 \
    --runs 5 \
    path/to/skill-or-plugin

This command triggers the full pipeline defined in EvaluateCommand.Run, including discovery, three-arm evaluation, judging, and multi-format reporting.

Reusing Cached Baselines

Avoid recomputing baseline runs by using the caching mechanism:

# First run – writes baseline data

skill-validator evaluate \
    --baseline-out baseline.json \
    --tests-dir tests \
    ...

# Later run – reads baseline instead of recomputing

skill-validator evaluate \
    --baseline-from baseline.json \
    --tests-dir tests \
    ...

The BaselineStore.cs handles persistence and retrieval of baseline runs via the BaselineOut and BaselineFrom configuration options.

Programmatic Integration

Integrate the evaluator directly into .NET applications using the EvaluateCommand.Run method:

using SkillValidator.Evaluate;

// Build a config the same way the CLI does
var config = new ValidatorConfig
{
    TestsDir = "tests",
    ResultsDir = ".skill-validator-results",
    Model = "claude-opus-4.6",
    Runs = 3,
    // …set any other options you need
};

// Run the evaluation and get the verdict objects back
var exitCode = await EvaluateCommand.Run(config);

This pattern allows the skill-validator evaluation framework architecture to be embedded in larger test suites or CI/CD pipelines.

Generating Markdown Reports

Extract human-readable summaries from evaluation results:

var verdicts = /* result from EvaluateCommand.Run */;
var markdown = Reporter.GenerateMarkdownSummary(verdicts, "claude-opus-4.6", "claude-opus-4.6");
Console.WriteLine(markdown);

The Reporter.GenerateMarkdownSummary method in Reporter.cs creates formatted markdown summaries suitable for documentation or pull request comments.

Summary

• The skill-validator evaluation framework architecture follows a modular pipeline design with clear separation between CLI handling, discovery, evaluation, judging, and reporting concerns.
• All source code resides under eng/skill-validator/src/ with Program.cs serving as the entry point and EvaluateCommand.Run as the main execution coordinator.
• The framework implements a three-arm evaluation strategy (baseline, isolated, plugin) using AgentRunner and ConcurrencyLimiter for parallel LLM execution.
• Baseline caching via BaselineStore.cs enables efficient reuse of baseline runs across evaluation sessions.
• Output formats include console, JSON, JUnit XML, and Markdown through the Reporter.cs class.

Frequently Asked Questions

How does the skill-validator handle parallel execution?

The framework uses ConcurrencyLimiter in the Shared/ directory to manage parallel LLM calls during evaluation. This component prevents API rate limiting while maximizing throughput across multiple test scenarios.

What is the purpose of the three-arm evaluation strategy?

The three-arm strategy implemented in EvaluateSkill.cs and EvaluateAgent.cs compares baseline performance (no skill), isolated skill execution, and plugin-mode execution. This isolates the true impact of the skill being evaluated from confounding variables and environment noise.

Can I run evaluations without recomputing baselines every time?

Yes. The BaselineStore.cs implements caching through --baseline-out and --baseline-from parameters. Store baseline results to a JSON file and reuse them across subsequent evaluation runs to save time and API costs.

How does the judging system prevent overfitting?

The OverfittingJudge.cs specifically analyzes LLM judgment patterns to detect when the judge model is overfitting to evaluation prompts rather than assessing actual skill quality. This works alongside the standard Judge.cs and Comparator.cs components to ensure valid statistical comparisons.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}