# How to Test Skills Before Deploying Them to Production: A 5-Stage Validation Guide

> Learn how to test skills before deploying to production using a 5-stage validation guide. Ensure metadata integrity, script execution, and functional behavior. Avoid production errors.

- Repository: [Anthropic/skills](https://github.com/anthropics/skills)
- Tags: how-to-guide
- Published: 2026-02-16

---

**Testing skills before deploying them to production requires a five-stage validation process that checks metadata integrity, script execution, functional behavior, and UI artifacts against the Agent Skills specification.**

The `anthropics/skills` repository provides a structured framework for building Claude-compatible skills. Before deploying these skills to production, you must validate them against the formal Agent Skills specification to ensure compatibility with Claude's runtime environment.

## The Five Stages of Production Skill Testing

Testing a skill is a multi-stage process that ensures the **metadata**, **runtime scripts**, and **behavior** of the skill all meet the Agent Skills specification before the skill is uploaded to Claude's production environment.

### Stage 1: Front-Matter Validation

The first stage validates that [`SKILL.md`](https://github.com/anthropics/skills/blob/main/SKILL.md) contains a proper YAML front-matter block with required fields: `name`, `description`, optional `license`, `allowed-tools`, `metadata`, and `compatibility`. The validator enforces naming conventions (kebab-case, ≤64 characters) and description length limits according to the **Agent Skills spec** located at [`spec/agent-skills-spec.md`](https://github.com/anthropics/skills/blob/main/spec/agent-skills-spec.md).

This validation is implemented in [`skills/skill-creator/scripts/quick_validate.py`](https://github.com/anthropics/skills/blob/main/skills/skill-creator/scripts/quick_validate.py), which parses [`SKILL.md`](https://github.com/anthropics/skills/blob/main/SKILL.md) and reports any violations.

```bash
python skills/skill-creator/scripts/quick_validate.py path/to/your/skill

```

If validation passes, the script outputs `Skill is valid!`. If the front-matter is malformed, it returns specific errors such as `Unexpected key(s) in SKILL.md frontmatter: foo, bar`.

### Stage 2: Unit-Style Script Testing

The second stage executes any helper scripts shipped with the skill—such as Playwright test scripts or data-processing helpers—to verify they run without errors and produce expected files. Scripts are stored under `skills/<skill-name>/scripts/`.

For example, the *webapp-testing* skill includes [`skills/webapp-testing/scripts/with_server.py`](https://github.com/anthropics/skills/blob/main/skills/webapp-testing/scripts/with_server.py), which starts a local server, waits for it to be reachable, runs the test script, then shuts the server down.

```bash
python skills/webapp-testing/scripts/with_server.py \
  --server "npm start" \
  --port 3000 \
  -- python test.py

```

### Stage 3: Functional Integration Testing

The third stage runs a **skill-harness** that simulates a Claude request, invokes the skill, and checks the output against an expected pattern (JSON, image, PDF, etc.). The repository provides a generic evaluation harness for MCP servers in [`skills/mcp-builder/scripts/evaluation.py`](https://github.com/anthropics/skills/blob/main/skills/mcp-builder/scripts/evaluation.py), which can be repurposed for any skill that returns structured data.

The harness requires an evaluation file (such as [`skills/mcp-builder/scripts/example_evaluation.xml`](https://github.com/anthropics/skills/blob/main/skills/mcp-builder/scripts/example_evaluation.xml)) and a questions JSON file.

```bash
python skills/mcp-builder/scripts/evaluation.py \
  --skill path/to/skill \
  --questions examples/eval_questions.json

```

The harness prints a pass/fail report for each question and saves full JSON responses to `evaluation_output/`.

### Stage 4: Manual UI Verification (Optional)

For skills that generate UI artifacts—such as web-app testing or visual artifact builders—you can spin up a temporary server and manually verify the artifact in a browser. The **Playwright**-based tests in the *webapp-testing* skill demonstrate this workflow.

Documentation for this process is located in [`skills/webapp-testing/SKILL.md`](https://github.com/anthropics/skills/blob/main/skills/webapp-testing/SKILL.md), which describes the testing workflow and references the Playwright scripts. Follow the "Run locally" instructions in that file to perform manual verification.

### Stage 5: Final Sanity Check

After automated checks pass, run the *skill-creator* quick-validation again and confirm the skill folder contains only the required files: [`SKILL.md`](https://github.com/anthropics/skills/blob/main/SKILL.md), `scripts/`, and any static assets. This ensures no stray files or metadata slipped in during development.

```bash
python skills/skill-creator/scripts/quick_validate.py path/to/skill

```

## Automating Your Testing Pipeline

You can combine all five stages into a continuous integration pipeline using a bash script. The following example validates front-matter for every skill, runs unit tests, and executes the functional evaluation harness:

```bash
#!/usr/bin/env bash
set -euo pipefail

# 1️⃣ Validate front-matter for every skill

for d in skills/*/ ; do
  python skills/skill-creator/scripts/quick_validate.py "$d"
done

# 2️⃣ Run any bundled scripts that have a "test.py"

find skills -name test.py -exec sh -c '
  dir=$(dirname "{}")
  echo "Running test in $dir"
  (cd "$dir" && python test.py)
' \;

# 3️⃣ Run the generic evaluation harness on a sample skill

python skills/mcp-builder/scripts/evaluation.py \
  --skill skills/pptx \
  --questions docs/eval_questions.json

```

## Key Testing Files in the Repository

| File | Role | Location |
|------|------|------------|
| [`quick_validate.py`](https://github.com/anthropics/skills/blob/main/quick_validate.py) | Front-matter validator used for every skill | [`skills/skill-creator/scripts/quick_validate.py`](https://github.com/anthropics/skills/blob/main/skills/skill-creator/scripts/quick_validate.py) |
| [`with_server.py`](https://github.com/anthropics/skills/blob/main/with_server.py) | Helper to start a local server and run Playwright-based tests | [`skills/webapp-testing/scripts/with_server.py`](https://github.com/anthropics/skills/blob/main/skills/webapp-testing/scripts/with_server.py) |
| [`evaluation.py`](https://github.com/anthropics/skills/blob/main/evaluation.py) | Generic evaluation harness for functional testing | [`skills/mcp-builder/scripts/evaluation.py`](https://github.com/anthropics/skills/blob/main/skills/mcp-builder/scripts/evaluation.py) |
| [`SKILL.md`](https://github.com/anthropics/skills/blob/main/SKILL.md) (webapp-testing) | Documentation of the web-app testing skill, includes testing instructions | [`skills/webapp-testing/SKILL.md`](https://github.com/anthropics/skills/blob/main/skills/webapp-testing/SKILL.md) |
| [`SKILL.md`](https://github.com/anthropics/skills/blob/main/SKILL.md) (skill-creator) | Specification of how a skill should be structured, including testing recommendations | [`skills/skill-creator/SKILL.md`](https://github.com/anthropics/skills/blob/main/skills/skill-creator/SKILL.md) |
| [`agent-skills-spec.md`](https://github.com/anthropics/skills/blob/main/agent-skills-spec.md) | The formal Agent Skills specification that the validator enforces | [`spec/agent-skills-spec.md`](https://github.com/anthropics/skills/blob/main/spec/agent-skills-spec.md) |

## Summary

- **Front-matter validation** ensures your [`SKILL.md`](https://github.com/anthropics/skills/blob/main/SKILL.md) metadata conforms to the Agent Skills specification using [`quick_validate.py`](https://github.com/anthropics/skills/blob/main/quick_validate.py).
- **Unit-style testing** executes helper scripts to catch runtime errors before deployment.
- **Functional integration testing** uses the [`evaluation.py`](https://github.com/anthropics/skills/blob/main/evaluation.py) harness to verify skill outputs match expected schemas.
- **Manual UI verification** provides a final visual check for skills generating web artifacts or images.
- **Automated pipelines** can combine all stages using bash scripts to prevent broken skills from reaching production.

## Frequently Asked Questions

### What happens if my skill fails front-matter validation?

If your skill fails front-matter validation, the [`quick_validate.py`](https://github.com/anthropics/skills/blob/main/quick_validate.py) script will output specific error messages indicating which required fields are missing or malformed. For example, it will report unexpected keys, naming convention violations (such as names exceeding 64 characters or not using kebab-case), or description length issues. You must fix these errors before the skill can load in Claude's environment.

### Can I test skills that require external services or APIs?

Yes, you can test skills that require external services using the **unit-style script testing** approach. The [`with_server.py`](https://github.com/anthropics/skills/blob/main/with_server.py) helper script is designed to manage external dependencies by starting local servers (such as npm development servers) before executing your test scripts. For external APIs, ensure your test scripts mock or safely interact with these services to prevent accidental data modification during testing.

### How does the functional integration harness simulate Claude requests?

The functional integration harness in [`skills/mcp-builder/scripts/evaluation.py`](https://github.com/anthropics/skills/blob/main/skills/mcp-builder/scripts/evaluation.py) simulates Claude requests by loading your skill and invoking it with predefined questions from a JSON evaluation file. It mimics the production runtime environment by processing the skill's output and comparing it against expected patterns, such as specific JSON schemas, image formats, or PDF structures. This ensures the skill's contract with Claude remains valid before deployment.

### Is manual UI verification necessary for all skills?

No, manual UI verification is only necessary for skills that generate **visual artifacts** such as web applications, images, or documents requiring subjective visual assessment. Skills that process data or return text responses can rely solely on automated validation stages. The manual verification stage uses Playwright-based workflows documented in [`skills/webapp-testing/SKILL.md`](https://github.com/anthropics/skills/blob/main/skills/webapp-testing/SKILL.md) to inspect rendered output in a controlled browser environment.