# How to Test Plugins Developed for dotnet/skills: The Complete Validation Guide

> Master testing dotnet/skills plugins! Learn to effectively validate your plugins using the skill-validator CLI and eval.yaml test scenarios in a sandboxed CI environment. Ensure robust skills development.

- Repository: [.NET Platform/skills](https://github.com/dotnet/skills)
- Tags: tutorial
- Published: 2026-05-22

---

**The dotnet/skills repository validates plugins using a skill-validator CLI that executes test scenarios defined in [`eval.yaml`](https://github.com/dotnet/skills/blob/main/eval.yaml) files against [`SKILL.md`](https://github.com/dotnet/skills/blob/main/SKILL.md) definitions within a sandboxed CI environment.**

Testing plugins in the dotnet/skills ecosystem involves a structured workflow that treats each plugin as a self-contained package bundling AI skills with automated validation. The repository uses a custom **skill-validator** binary and GitHub Actions to orchestrate discovery, building, evaluation, and reporting. This guide covers the exact file paths, configuration schemas, and commands needed to validate your plugins both locally and in continuous integration.

## Understanding the Testing Architecture

The dotnet/skills testing framework relies on a specific directory structure that separates plugin definitions from their test scenarios. The **skill-validator**—located in `eng/skill-validator/src/**`—compiles into a CLI tool that reads skill definitions and executes assertions against them.

Key components include:

- **`plugins/<plugin>/plugin.json`** – The plugin manifest declaring the plugin name, version, and skills folder location
- **`plugins/<plugin>/skills/<skill>/SKILL.md`** – Human-readable skill descriptions with inputs and metadata
- **`tests/<plugin>/<skill>/eval.yaml`** – Test specifications defining evaluation scenarios and expected outputs
- **[`.github/workflows/evaluation.yml`](https://github.com/dotnet/skills/blob/main/.github/workflows/evaluation.yml)** – The CI workflow that detects changes and orchestrates validation

The validator operates in a sandboxed environment using the [`global.json`](https://github.com/dotnet/skills/blob/main/global.json) SDK version, ensuring deterministic test results that reflect the exact plugin state at the evaluated commit.

## Configuring Test Scenarios with eval.yaml

The [`eval.yaml`](https://github.com/dotnet/skills/blob/main/eval.yaml) file defines how the skill-validator exercises your skill. Located at `tests/<plugin>/<skill>/eval.yaml`, this configuration specifies inputs, expected outputs, and prompt templates.

### Schema Structure

A minimal [`eval.yaml`](https://github.com/dotnet/skills/blob/main/eval.yaml) contains:

```yaml
inputs:
  - name: code
    description: C# snippet to refactor

    type: string
expected:
  - type: string
    contains: "refactored"
scenario:
  prompt: |
    Refactor the following code:
    {{code}}

```

The validator replaces template variables like `{{code}}` with test inputs and asserts that the model's response meets the specified conditions. For concrete test values, use the `value` field instead of `description`:

```yaml
inputs:
  - name: code
    value: |
      var x = 1;
      Console.WriteLine(x);
expected:
  - type: string
    contains: "var"

```

## The CI/CD Evaluation Workflow

The [`.github/workflows/evaluation.yml`](https://github.com/dotnet/skills/blob/main/.github/workflows/evaluation.yml) file automates plugin testing through a multi-job pipeline that ensures security and reproducibility.

### Workflow Stages

1. **`pr-status`** – Posts a pending status check when skill files change in a PR
2. **`gate`** – Validates maintainer permissions when `/evaluate` is commented
3. **`discover`** (lines 91-120) – Scans for files matching `plugins/*/skills/*/SKILL.md` and corresponding `tests/*` directories, building a matrix of `(plugin, skill)` pairs to evaluate
4. **`build-validator`** – Compiles the skill-validator binary from the current commit for same-repo PRs, or from the base branch for fork PRs
5. **`evaluate`** – Executes the validator against each matrix entry using the test definitions
6. **`comment-on-pr`** (lines 61-78) – Consolidates JSON results into a markdown table and posts a summary comment

### Security Isolation

The workflow implements strict security boundaries. For pull requests from forks, the validator binary builds from the base branch to prevent untrusted code execution. Same-repository PRs build the validator from the PR branch itself, allowing you to test changes to the validator logic simultaneously.

### Triggering Evaluations

After opening a PR with plugin changes, a maintainer must comment `/evaluate` to trigger the evaluation. The `gate` job verifies the commenter's permissions before allowing the `discover` job to proceed. The workflow also automatically detects test-only changes and can trigger without explicit commands in those cases.

## Local Testing Workflow

You can validate plugins locally before submitting a PR using the **skill-installer** CLI to invoke the validator directly.

### Installing the Validator

Install the validator from the repository source:

```bash
skill-installer install https://github.com/dotnet/skills/tree/main/eng/skill-validator/src

```

### Executing Tests Locally

Run the validator against a specific plugin and skill:

```bash
skill-validator evaluate \
  --tests-dir ./tests/my-plugin \
  --results-dir ./tmp/results \
  ./plugins/my-plugin/skills/my-skill

```

Replace `my-plugin` and `my-skill` with your actual directory names. The command produces JSON results at `./tmp/results/<plugin>--<skill>/results.json`, which you can inspect with `jq` or other tools. These files match the artifact format uploaded by CI, ensuring consistency between local and remote testing.

## Creating Tests for New Plugins

To establish a complete testable plugin, you must create both the skill definition and its corresponding test configuration.

### Step 1: Define the Plugin Structure

Create the plugin manifest and skill folder:

```text
plugins/my-plugin/
  plugin.json
  skills/
    my-skill/
      SKILL.md

```

The [`plugin.json`](https://github.com/dotnet/skills/blob/main/plugin.json) must declare the skills path:

```json
{
  "name": "my-plugin",
  "version": "0.1.0",
  "description": "Custom skills for demonstration.",
  "skills": ["./skills/"],
  "lspServers": "./lsp.json"
}

```

### Step 2: Document the Skill

Create [`SKILL.md`](https://github.com/dotnet/skills/blob/main/SKILL.md) with metadata and input definitions:

```markdown
name: my-skill
description: |
  Refactors a given C# snippet to use modern language features.

inputs:
  - name: code
    description: The C# code to refactor.

    type: string

```

### Step 3: Add Test Scenarios

Mirror the plugin structure under `tests/`:

```text
tests/my-plugin/
  my-skill/
    eval.yaml

```

Populate [`eval.yaml`](https://github.com/dotnet/skills/blob/main/eval.yaml) with specific test cases that the validator can execute against your skill's logic.

### Step 4: Validate and Submit

Commit your changes and open a PR. The system will detect the new skill and test files. After a maintainer comments `/evaluate`, the workflow will execute your tests and post results directly to the PR as a markdown table showing Pass/Fail status per skill.

## Summary

- **Test configuration** resides in `tests/<plugin>/<skill>/eval.yaml` and defines inputs, expected outputs, and prompt scenarios for the skill-validator.
- **CI automation** runs via [`.github/workflows/evaluation.yml`](https://github.com/dotnet/skills/blob/main/.github/workflows/evaluation.yml), which discovers changed skills, builds the validator, and posts results to PRs after a maintainer triggers evaluation with `/evaluate`.
- **Local validation** uses the `skill-validator evaluate` command with `--tests-dir` and `--results-dir` arguments to execute the same tests locally that run in CI.
- **Security model** uses base-branch validators for fork PRs and PR-branch validators for same-repo contributions, preventing arbitrary code execution while allowing validator development.
- **Required files** include [`plugin.json`](https://github.com/dotnet/skills/blob/main/plugin.json) for metadata, [`SKILL.md`](https://github.com/dotnet/skills/blob/main/SKILL.md) for skill definitions, and [`eval.yaml`](https://github.com/dotnet/skills/blob/main/eval.yaml) for test specifications.

## Frequently Asked Questions

### How do I trigger plugin tests in a pull request?

After pushing changes to your PR, request a maintainer to comment `/evaluate`. The workflow defined in [`.github/workflows/evaluation.yml`](https://github.com/dotnet/skills/blob/main/.github/workflows/evaluation.yml) will detect this command, verify permissions through the `gate` job, and execute the `discover` job to find and test your modified plugins. The results appear as a markdown table in the PR comments.

### Can I run dotnet/skills plugin tests on my local machine?

Yes. Install the validator using `skill-installer install` pointing to `eng/skill-validator/src`, then execute `skill-validator evaluate` with the `--tests-dir` and `--results-dir` flags. This runs the same evaluation logic used in CI against your local plugin and skill definitions.

### What is the relationship between SKILL.md and eval.yaml files?

[`SKILL.md`](https://github.com/dotnet/skills/blob/main/SKILL.md) files located at `plugins/<plugin>/skills/<skill>/SKILL.md` document the skill's purpose, inputs, and behavior for human readers and the AI system. The [`eval.yaml`](https://github.com/dotnet/skills/blob/main/eval.yaml) files at `tests/<plugin>/<skill>/eval.yaml` define automated test scenarios that validate the skill's actual outputs against expected results using the skill-validator CLI.

### How does the evaluation workflow handle security for external contributions?

For pull requests from forks, the `build-validator` job compiles the validator binary from the repository's base branch rather than the PR branch. This prevents untrusted code from executing arbitrary commands in the CI environment. Same-repository PRs build the validator from the PR branch itself, allowing you to test modifications to the validator source code safely.