# Skill-Validator Evaluation Framework Architecture: Inside the .NET Pipeline

> Discover the skill-validator evaluation framework architecture. Explore its .NET pipeline for AI skill discovery, LLM evaluations, and quality reports.

- Repository: [.NET Platform/skills](https://github.com/dotnet/skills)
- Tags: architecture
- Published: 2026-07-07

---

**The skill-validator evaluation framework architecture implements a modular command-line pipeline that discovers AI skills, executes three-arm LLM evaluations, and produces statistical quality reports via a self-contained .NET application.**

The dotnet/skills repository hosts a sophisticated evaluation system designed to validate AI agents and skills through automated testing. This self-contained .NET console application orchestrates complex workflows including baseline comparisons, isolated testing, and plugin integration. Understanding the skill-validator evaluation framework architecture reveals how the system handles filesystem discovery, parallel LLM execution, statistical analysis, and multi-format reporting.

## Architecture Layers

The framework organizes code into distinct logical layers under `eng/skill-validator/src/`, each with specific responsibilities and clear separation of concerns.

### CLI Entry Point and Command Layer

Located in [`Program.cs`](https://github.com/dotnet/skills/blob/main/Program.cs), this layer initializes the **System.CommandLine** infrastructure to create a `RootCommand` with `evaluate` and `check` subcommands. The [`Evaluate/EvaluateCommand.cs`](https://github.com/dotnet/skills/blob/main/Evaluate/EvaluateCommand.cs) and [`Check/CheckCommand.cs`](https://github.com/dotnet/skills/blob/main/Check/CheckCommand.cs) files parse command-line arguments and translate them into a `ValidatorConfig` object that drives the entire pipeline.

### Discovery Layer

The [`Shared/SkillDiscovery.cs`](https://github.com/dotnet/skills/blob/main/Shared/SkillDiscovery.cs), [`Shared/PluginDiscovery.cs`](https://github.com/dotnet/skills/blob/main/Shared/PluginDiscovery.cs), and [`Shared/AgentDiscovery.cs`](https://github.com/dotnet/skills/blob/main/Shared/AgentDiscovery.cs) files handle filesystem traversal to locate [`plugin.json`](https://github.com/dotnet/skills/blob/main/plugin.json), [`skill.yml`](https://github.com/dotnet/skills/blob/main/skill.yml), and `*.agent.md` files. This layer produces `SkillInfo`, `AgentInfo`, and plugin-root path objects consumed by the evaluation engine.

### Evaluation Engine

The core orchestration lives in `EvaluateCommand.Run`, `EvaluateTarget`, `EvaluateSkill`, and `EvaluateAgent`. This engine implements the **three-arm evaluation** strategy (baseline, isolated, plugin) using `AgentRunner` for LLM execution, `ConcurrencyLimiter` for parallelism control, and `RetryHelper` for resilience. The [`Shared/Statistics.cs`](https://github.com/dotnet/skills/blob/main/Shared/Statistics.cs) and [`Shared/Models.cs`](https://github.com/dotnet/skills/blob/main/Shared/Models.cs) files define the data structures for metrics collection.

### Judging and Comparison

Quality assessment occurs in [`Evaluate/Judge.cs`](https://github.com/dotnet/skills/blob/main/Evaluate/Judge.cs), which calls the LLM judge model to generate rubric scores. The [`Comparator.cs`](https://github.com/dotnet/skills/blob/main/Comparator.cs) handles pairwise comparisons when `--judge-mode=pairwise` is specified, while [`OverfittingJudge.cs`](https://github.com/dotnet/skills/blob/main/OverfittingJudge.cs) detects judgment overfitting to evaluation prompts. These components produce `ScenarioComparison` objects that feed into the final `SkillVerdict`.

### Baseline Caching

The [`BaselineStore.cs`](https://github.com/dotnet/skills/blob/main/BaselineStore.cs) implements a caching mechanism for the baseline arm (no-skill/no-agent execution). This allows repeated runs to reuse baseline data via `--baseline-from` and `--baseline-out` parameters, significantly reducing evaluation time and API costs.

### Reporting Layer

The [`Reporter.cs`](https://github.com/dotnet/skills/blob/main/Reporter.cs) class formats results across multiple output formats including console, JSON, JUnit XML, and Markdown. It provides `GenerateMarkdownSummary` for human-readable reports and handles per-scenario markdown generation.

## Data Flow Through the Pipeline

Execution begins at [`Program.cs`](https://github.com/dotnet/skills/blob/main/Program.cs) with command parsing, flows through the discovery layer to build in-memory models, then enters the evaluation engine. The `ValidatorConfig` acts as the central configuration object passed through each stage. `AgentRunner` executes LLM calls while `Judge` and `Comparator` analyze outputs and compute improvement scores against the baseline. Finally, `Reporter` formats the list of `SkillVerdict` objects into the requested output formats.

## Practical Usage Examples

### Running a Full Evaluation

Invoke the evaluation pipeline from the command line using the `evaluate` subcommand:

```bash
skill-validator evaluate \
    --tests-dir tests \
    --results-dir .skill-validator-results \
    --reporter console json markdown \
    --model claude-opus-4.6 \
    --runs 5 \
    path/to/skill-or-plugin

```

This command triggers the full pipeline defined in `EvaluateCommand.Run`, including discovery, three-arm evaluation, judging, and multi-format reporting.

### Reusing Cached Baselines

Avoid recomputing baseline runs by using the caching mechanism:

```bash

# First run – writes baseline data

skill-validator evaluate \
    --baseline-out baseline.json \
    --tests-dir tests \
    ...

# Later run – reads baseline instead of recomputing

skill-validator evaluate \
    --baseline-from baseline.json \
    --tests-dir tests \
    ...

```

The [`BaselineStore.cs`](https://github.com/dotnet/skills/blob/main/BaselineStore.cs) handles persistence and retrieval of baseline runs via the `BaselineOut` and `BaselineFrom` configuration options.

### Programmatic Integration

Integrate the evaluator directly into .NET applications using the `EvaluateCommand.Run` method:

```csharp
using SkillValidator.Evaluate;

// Build a config the same way the CLI does
var config = new ValidatorConfig
{
    TestsDir = "tests",
    ResultsDir = ".skill-validator-results",
    Model = "claude-opus-4.6",
    Runs = 3,
    // …set any other options you need
};

// Run the evaluation and get the verdict objects back
var exitCode = await EvaluateCommand.Run(config);

```

This pattern allows the skill-validator evaluation framework architecture to be embedded in larger test suites or CI/CD pipelines.

### Generating Markdown Reports

Extract human-readable summaries from evaluation results:

```csharp
var verdicts = /* result from EvaluateCommand.Run */;
var markdown = Reporter.GenerateMarkdownSummary(verdicts, "claude-opus-4.6", "claude-opus-4.6");
Console.WriteLine(markdown);

```

The `Reporter.GenerateMarkdownSummary` method in [`Reporter.cs`](https://github.com/dotnet/skills/blob/main/Reporter.cs) creates formatted markdown summaries suitable for documentation or pull request comments.

## Summary

- The skill-validator evaluation framework architecture follows a modular pipeline design with clear separation between CLI handling, discovery, evaluation, judging, and reporting concerns.
- All source code resides under `eng/skill-validator/src/` with [`Program.cs`](https://github.com/dotnet/skills/blob/main/Program.cs) serving as the entry point and `EvaluateCommand.Run` as the main execution coordinator.
- The framework implements a three-arm evaluation strategy (baseline, isolated, plugin) using `AgentRunner` and `ConcurrencyLimiter` for parallel LLM execution.
- Baseline caching via [`BaselineStore.cs`](https://github.com/dotnet/skills/blob/main/BaselineStore.cs) enables efficient reuse of baseline runs across evaluation sessions.
- Output formats include console, JSON, JUnit XML, and Markdown through the [`Reporter.cs`](https://github.com/dotnet/skills/blob/main/Reporter.cs) class.

## Frequently Asked Questions

### How does the skill-validator handle parallel execution?

The framework uses `ConcurrencyLimiter` in the `Shared/` directory to manage parallel LLM calls during evaluation. This component prevents API rate limiting while maximizing throughput across multiple test scenarios.

### What is the purpose of the three-arm evaluation strategy?

The three-arm strategy implemented in [`EvaluateSkill.cs`](https://github.com/dotnet/skills/blob/main/EvaluateSkill.cs) and [`EvaluateAgent.cs`](https://github.com/dotnet/skills/blob/main/EvaluateAgent.cs) compares baseline performance (no skill), isolated skill execution, and plugin-mode execution. This isolates the true impact of the skill being evaluated from confounding variables and environment noise.

### Can I run evaluations without recomputing baselines every time?

Yes. The [`BaselineStore.cs`](https://github.com/dotnet/skills/blob/main/BaselineStore.cs) implements caching through `--baseline-out` and `--baseline-from` parameters. Store baseline results to a JSON file and reuse them across subsequent evaluation runs to save time and API costs.

### How does the judging system prevent overfitting?

The [`OverfittingJudge.cs`](https://github.com/dotnet/skills/blob/main/OverfittingJudge.cs) specifically analyzes LLM judgment patterns to detect when the judge model is overfitting to evaluation prompts rather than assessing actual skill quality. This works alongside the standard [`Judge.cs`](https://github.com/dotnet/skills/blob/main/Judge.cs) and [`Comparator.cs`](https://github.com/dotnet/skills/blob/main/Comparator.cs) components to ensure valid statistical comparisons.