How to Test Plugins Developed for dotnet/skills: The Complete Validation Guide

The dotnet/skills repository validates plugins using a skill-validator CLI that executes test scenarios defined in eval.yaml files against SKILL.md definitions within a sandboxed CI environment.

Testing plugins in the dotnet/skills ecosystem involves a structured workflow that treats each plugin as a self-contained package bundling AI skills with automated validation. The repository uses a custom skill-validator binary and GitHub Actions to orchestrate discovery, building, evaluation, and reporting. This guide covers the exact file paths, configuration schemas, and commands needed to validate your plugins both locally and in continuous integration.

Understanding the Testing Architecture

The dotnet/skills testing framework relies on a specific directory structure that separates plugin definitions from their test scenarios. The skill-validator—located in eng/skill-validator/src/**—compiles into a CLI tool that reads skill definitions and executes assertions against them.

Key components include:

• plugins/<plugin>/plugin.json – The plugin manifest declaring the plugin name, version, and skills folder location
• plugins/<plugin>/skills/<skill>/SKILL.md – Human-readable skill descriptions with inputs and metadata
• tests/<plugin>/<skill>/eval.yaml – Test specifications defining evaluation scenarios and expected outputs
• .github/workflows/evaluation.yml – The CI workflow that detects changes and orchestrates validation

The validator operates in a sandboxed environment using the global.json SDK version, ensuring deterministic test results that reflect the exact plugin state at the evaluated commit.

Configuring Test Scenarios with eval.yaml

The eval.yaml file defines how the skill-validator exercises your skill. Located at tests/<plugin>/<skill>/eval.yaml, this configuration specifies inputs, expected outputs, and prompt templates.

Schema Structure

A minimal eval.yaml contains:

inputs:
  - name: code
    description: C# snippet to refactor

    type: string
expected:
  - type: string
    contains: "refactored"
scenario:
  prompt: |
    Refactor the following code:
    {{code}}

The validator replaces template variables like {{code}} with test inputs and asserts that the model's response meets the specified conditions. For concrete test values, use the value field instead of description:

inputs:
  - name: code
    value: |
      var x = 1;
      Console.WriteLine(x);
expected:
  - type: string
    contains: "var"

The CI/CD Evaluation Workflow

The .github/workflows/evaluation.yml file automates plugin testing through a multi-job pipeline that ensures security and reproducibility.

Workflow Stages

1. pr-status – Posts a pending status check when skill files change in a PR
2. gate – Validates maintainer permissions when /evaluate is commented
3. discover (lines 91-120) – Scans for files matching plugins/*/skills/*/SKILL.md and corresponding tests/* directories, building a matrix of (plugin, skill) pairs to evaluate
4. build-validator – Compiles the skill-validator binary from the current commit for same-repo PRs, or from the base branch for fork PRs
5. evaluate – Executes the validator against each matrix entry using the test definitions
6. comment-on-pr (lines 61-78) – Consolidates JSON results into a markdown table and posts a summary comment

Security Isolation

The workflow implements strict security boundaries. For pull requests from forks, the validator binary builds from the base branch to prevent untrusted code execution. Same-repository PRs build the validator from the PR branch itself, allowing you to test changes to the validator logic simultaneously.

Triggering Evaluations

After opening a PR with plugin changes, a maintainer must comment /evaluate to trigger the evaluation. The gate job verifies the commenter's permissions before allowing the discover job to proceed. The workflow also automatically detects test-only changes and can trigger without explicit commands in those cases.

Local Testing Workflow

You can validate plugins locally before submitting a PR using the skill-installer CLI to invoke the validator directly.

Installing the Validator

Install the validator from the repository source:

skill-installer install https://github.com/dotnet/skills/tree/main/eng/skill-validator/src

Executing Tests Locally

Run the validator against a specific plugin and skill:

skill-validator evaluate \
  --tests-dir ./tests/my-plugin \
  --results-dir ./tmp/results \
  ./plugins/my-plugin/skills/my-skill

Replace my-plugin and my-skill with your actual directory names. The command produces JSON results at ./tmp/results/<plugin>--<skill>/results.json, which you can inspect with jq or other tools. These files match the artifact format uploaded by CI, ensuring consistency between local and remote testing.

Creating Tests for New Plugins

To establish a complete testable plugin, you must create both the skill definition and its corresponding test configuration.

Step 1: Define the Plugin Structure

Create the plugin manifest and skill folder:

plugins/my-plugin/
  plugin.json
  skills/
    my-skill/
      SKILL.md

The plugin.json must declare the skills path:

{
  "name": "my-plugin",
  "version": "0.1.0",
  "description": "Custom skills for demonstration.",
  "skills": ["./skills/"],
  "lspServers": "./lsp.json"
}

Step 2: Document the Skill

Create SKILL.md with metadata and input definitions:

name: my-skill
description: |
  Refactors a given C# snippet to use modern language features.

inputs:
  - name: code
    description: The C# code to refactor.

    type: string

Step 3: Add Test Scenarios

Mirror the plugin structure under tests/:

tests/my-plugin/
  my-skill/
    eval.yaml

Populate eval.yaml with specific test cases that the validator can execute against your skill's logic.

Step 4: Validate and Submit

Commit your changes and open a PR. The system will detect the new skill and test files. After a maintainer comments /evaluate, the workflow will execute your tests and post results directly to the PR as a markdown table showing Pass/Fail status per skill.

Summary

• Test configuration resides in tests/<plugin>/<skill>/eval.yaml and defines inputs, expected outputs, and prompt scenarios for the skill-validator.
• CI automation runs via .github/workflows/evaluation.yml, which discovers changed skills, builds the validator, and posts results to PRs after a maintainer triggers evaluation with /evaluate.
• Local validation uses the skill-validator evaluate command with --tests-dir and --results-dir arguments to execute the same tests locally that run in CI.
• Security model uses base-branch validators for fork PRs and PR-branch validators for same-repo contributions, preventing arbitrary code execution while allowing validator development.
• Required files include plugin.json for metadata, SKILL.md for skill definitions, and eval.yaml for test specifications.

Frequently Asked Questions

How do I trigger plugin tests in a pull request?

After pushing changes to your PR, request a maintainer to comment /evaluate. The workflow defined in .github/workflows/evaluation.yml will detect this command, verify permissions through the gate job, and execute the discover job to find and test your modified plugins. The results appear as a markdown table in the PR comments.

Can I run dotnet/skills plugin tests on my local machine?

Yes. Install the validator using skill-installer install pointing to eng/skill-validator/src, then execute skill-validator evaluate with the --tests-dir and --results-dir flags. This runs the same evaluation logic used in CI against your local plugin and skill definitions.

What is the relationship between SKILL.md and eval.yaml files?

SKILL.md files located at plugins/<plugin>/skills/<skill>/SKILL.md document the skill's purpose, inputs, and behavior for human readers and the AI system. The eval.yaml files at tests/<plugin>/<skill>/eval.yaml define automated test scenarios that validate the skill's actual outputs against expected results using the skill-validator CLI.

How does the evaluation workflow handle security for external contributions?

For pull requests from forks, the build-validator job compiles the validator binary from the repository's base branch rather than the PR branch. This prevents untrusted code from executing arbitrary commands in the CI environment. Same-repository PRs build the validator from the PR branch itself, allowing you to test modifications to the validator source code safely.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}