Architectural Patterns in dotnet/skills: A Deep Dive into the Plug-In-Centric Design

The dotnet/skills repository implements a modular, command-driven architecture using plug-in discovery, strategy-based judges, and source-generated serialization to enable scalable skill validation.

The dotnet/skills codebase serves as a validation and evaluation engine for AI skills, employing sophisticated architectural patterns to maintain modularity and performance. This system separates concerns through a combination of discovery patterns, command encapsulation, and runtime strategy selection. Understanding these architectural patterns reveals how the repository manages hundreds of independent skills while maintaining clean dependencies and high throughput.

Plug-In Architecture and Service Discovery

The foundation of the system rests on a plug-in-centric architecture that treats skills and agents as independent modules. Each plug-in contains a plugin.json manifest describing its name, version, and paths to skill and agent files.

Manifest-Based Plugin Loading

The runtime discovers plug-ins by walking the directory tree and parsing these manifests. In eng/skill-validator/src/Shared/PluginDiscovery.cs, the discovery logic locates the plug-in root from any file path within the skill hierarchy:

// Given a file path inside a skill, locate the plug‑in root and parse its manifest.
var skillInfo = new SkillInfo { Path = "/path/to/skills/dotnet-test/skills/find-untested-sources/Skill.cs" };

var ctx = PluginDiscovery.FindPluginContext(skillInfo);
if (ctx is null)
{
    Console.WriteLine("No plug‑in found.");
    return;
}

var (pluginRoot, pluginInfo) = ctx;
Console.WriteLine($"Plugin: {pluginInfo.Name} (v{pluginInfo.Version})");
Console.WriteLine($"Root folder: {pluginRoot}");
Console.WriteLine("Skills declared in manifest:");
foreach (var skillPath in pluginInfo.SkillPaths)
    Console.WriteLine($"  - {skillPath}");

Reflection-Based Skill Discovery

Static helper classes decouple the core validator from concrete implementations. The SkillDiscovery and AgentDiscovery classes in eng/skill-validator/src/Shared/SkillDiscovery.cs use reflection to locate all skills belonging to a discovered plug-in, enforcing the Service Locator pattern without hard-coded dependencies.

Command Pattern for CLI Workflows

The CLI layer acts as a thin wrapper that delegates work to discrete command classes, implementing the Command pattern to encapsulate distinct workflows. Each command class handles a specific operation:

• CheckCommand (eng/skill-validator/src/Check/CheckCommand.cs) – Handles validation logic
• EvaluateCommand (eng/skill-validator/src/Evaluate/EvaluateCommand.cs) – Manages evaluation runs
• RejudgeCommand and ConsolidateCommand – Handle re-running and result aggregation

This structure allows the CLI to invoke complex operations through a uniform interface:

// The CLI invokes the command class directly.
var evalFile = "my-eval.yaml";
var options = new EvaluateOptions
{
    EvalFile = evalFile,
    MaxConcurrency = 4,
    EnableOverfittingCheck = true,
};

await EvaluateCommand.RunAsync(options);

Strategy Pattern for Evaluation Logic

The evaluation system employs the Strategy pattern to swap algorithms at runtime. Different judges implement a common IJudge interface defined in eng/skill-validator/src/Evaluate/Judge.cs, allowing the system to select evaluation strategies based on configuration options:

• OverfittingJudge – Detects overfitting in skill performance
• PairwiseJudge – Compares outputs pairwise
• MetricsCollector – Gathers performance metrics

Runtime selection occurs without modifying the evaluation engine core:

IJudge judge = options.UsePairwise
    ? new PairwiseJudge()
    : (IJudge)new OverfittingJudge();

var result = await judge.RunAsync(evaluationContext);

Builder, Factory, and Repository Patterns

Data management combines multiple patterns to ensure configuration flexibility and persistent state handling.

YAML Configuration Building

Evaluation configurations use a Builder pattern implementation in eng/skill-validator/src/Evaluate/EvalSchema.cs. The parsing logic creates strongly-typed models through EvalSchema.Parse, acting as a factory that transforms YAML files into executable evaluation plans.

File-Based Session Persistence

The SessionDatabase class in eng/skill-validator/src/Evaluate/SessionDatabase.cs implements the Repository and Unit-of-Work patterns for lightweight file-based storage. It manages metrics and session files while implementing IDisposable to ensure resources release when validation runs complete.

Concurrency Control and Resource Management

Parallel evaluation of multiple skills could exhaust host resources without proper throttling. The Concurrency-Limiter pattern wraps SemaphoreSlim in eng/skill-validator/src/Shared/ConcurrencyLimiter.cs to control parallelism:

using var limiter = new ConcurrencyLimiter(maxConcurrency: 8);
await Parallel.ForEachAsync(skills, async (skill, ct) =>
{
    await limiter.EnterAsync(ct);
    try
    {
        await EvaluateSkillAsync(skill);
    }
    finally
    {
        limiter.Release();
    }
});

This protects the system while maximizing throughput across CPU-bound validation tasks.

Advanced Patterns for Maintainability

Several additional patterns support code organization and performance optimization.

Source-Generated JSON Serialization

To avoid runtime reflection overhead, the repository uses System.Text.Json source generators. The SkillValidatorJsonContext class in eng/skill-validator/src/SkillValidatorJsonContext.cs provides compile-time-checked, high-performance JSON serialization for plugin.json files and result outputs.

Partial Classes for Separation of Concerns

Core services utilize the Partial-Class pattern to separate generated code from hand-written logic. For example, SkillValidatorYamlContext and OverfittingJudge split their implementations across multiple files, keeping source-generated serializers isolated from business logic.

Adapter and Decorator Patterns

• Adapter pattern: The run-vally-evals.sh script and adapt.mjs module adapt external benchmark outputs into the internal result model, bridging third-party tools with the validation engine.
• Decorator-like helpers: Utility types such as Spinner, Ansi, and RetryHelper in eng/skill-validator/src/Shared/Spinner.cs wrap cross-cutting concerns including progress UI, console coloring, and retry logic without polluting core business code.

Summary

The dotnet/skills repository demonstrates a mature application of enterprise architectural patterns:

• Plug-in Architecture enables independent skill development and discovery via manifest files
• Command Pattern encapsulates CLI workflows in discrete, testable classes
• Strategy Pattern allows runtime selection of evaluation algorithms through the IJudge interface
• Repository Pattern provides lightweight persistence through SessionDatabase
• Concurrency Limiter prevents resource exhaustion during parallel evaluation
• Source Generation eliminates runtime reflection costs for JSON serialization

These patterns collectively create a modular, extensible validation engine capable of scaling to hundreds of independent skills while maintaining clean architectural boundaries.

Frequently Asked Questions

What is the primary architectural style of dotnet/skills?

The repository follows a plug-in-centric, command-driven architectural style. Skills are self-contained plug-ins discovered at runtime through manifest files, while the CLI delegates operations to command objects that encapsulate specific workflows like validation and evaluation.

How does the repository handle parallel skill evaluation safely?

The codebase implements a Concurrency-Limiter pattern using SemaphoreSlim wrapped in the ConcurrencyLimiter class. This throttles parallel execution to a configurable maximum, preventing thread pool exhaustion while still allowing concurrent evaluation of independent skills.

Why does the codebase use source-generated JSON serialization instead of reflection?

The SkillValidatorJsonContext class provides source-generated JSON serialization using System.Text.Json generators. This approach offers compile-time validation of serialization contracts and eliminates runtime reflection overhead, significantly improving cold-start performance for the validation engine.

How are external benchmark tools integrated into the validation workflow?

The system uses the Adapter pattern through the run-vally-evals.sh shell script and adapt.mjs module. These components translate external benchmark outputs into the internal result model, allowing the core validation engine to consume third-party evaluation data without direct coupling to external tool implementations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}