How to Test Skills Before Deploying Them to Production: A 5-Stage Validation Guide

Testing skills before deploying them to production requires a five-stage validation process that checks metadata integrity, script execution, functional behavior, and UI artifacts against the Agent Skills specification.

The anthropics/skills repository provides a structured framework for building Claude-compatible skills. Before deploying these skills to production, you must validate them against the formal Agent Skills specification to ensure compatibility with Claude's runtime environment.

The Five Stages of Production Skill Testing

Testing a skill is a multi-stage process that ensures the metadata, runtime scripts, and behavior of the skill all meet the Agent Skills specification before the skill is uploaded to Claude's production environment.

Stage 1: Front-Matter Validation

The first stage validates that SKILL.md contains a proper YAML front-matter block with required fields: name, description, optional license, allowed-tools, metadata, and compatibility. The validator enforces naming conventions (kebab-case, ≤64 characters) and description length limits according to the Agent Skills spec located at spec/agent-skills-spec.md.

This validation is implemented in skills/skill-creator/scripts/quick_validate.py, which parses SKILL.md and reports any violations.

python skills/skill-creator/scripts/quick_validate.py path/to/your/skill

If validation passes, the script outputs Skill is valid!. If the front-matter is malformed, it returns specific errors such as Unexpected key(s) in SKILL.md frontmatter: foo, bar.

Stage 2: Unit-Style Script Testing

The second stage executes any helper scripts shipped with the skill—such as Playwright test scripts or data-processing helpers—to verify they run without errors and produce expected files. Scripts are stored under skills/<skill-name>/scripts/.

For example, the webapp-testing skill includes skills/webapp-testing/scripts/with_server.py, which starts a local server, waits for it to be reachable, runs the test script, then shuts the server down.

python skills/webapp-testing/scripts/with_server.py \
  --server "npm start" \
  --port 3000 \
  -- python test.py

Stage 3: Functional Integration Testing

The third stage runs a skill-harness that simulates a Claude request, invokes the skill, and checks the output against an expected pattern (JSON, image, PDF, etc.). The repository provides a generic evaluation harness for MCP servers in skills/mcp-builder/scripts/evaluation.py, which can be repurposed for any skill that returns structured data.

The harness requires an evaluation file (such as skills/mcp-builder/scripts/example_evaluation.xml) and a questions JSON file.

python skills/mcp-builder/scripts/evaluation.py \
  --skill path/to/skill \
  --questions examples/eval_questions.json

The harness prints a pass/fail report for each question and saves full JSON responses to evaluation_output/.

Stage 4: Manual UI Verification (Optional)

For skills that generate UI artifacts—such as web-app testing or visual artifact builders—you can spin up a temporary server and manually verify the artifact in a browser. The Playwright-based tests in the webapp-testing skill demonstrate this workflow.

Documentation for this process is located in skills/webapp-testing/SKILL.md, which describes the testing workflow and references the Playwright scripts. Follow the "Run locally" instructions in that file to perform manual verification.

Stage 5: Final Sanity Check

After automated checks pass, run the skill-creator quick-validation again and confirm the skill folder contains only the required files: SKILL.md, scripts/, and any static assets. This ensures no stray files or metadata slipped in during development.

python skills/skill-creator/scripts/quick_validate.py path/to/skill

Automating Your Testing Pipeline

You can combine all five stages into a continuous integration pipeline using a bash script. The following example validates front-matter for every skill, runs unit tests, and executes the functional evaluation harness:

#!/usr/bin/env bash
set -euo pipefail

# 1️⃣ Validate front-matter for every skill

for d in skills/*/ ; do
  python skills/skill-creator/scripts/quick_validate.py "$d"
done

# 2️⃣ Run any bundled scripts that have a "test.py"

find skills -name test.py -exec sh -c '
  dir=$(dirname "{}")
  echo "Running test in $dir"
  (cd "$dir" && python test.py)
' \;

# 3️⃣ Run the generic evaluation harness on a sample skill

python skills/mcp-builder/scripts/evaluation.py \
  --skill skills/pptx \
  --questions docs/eval_questions.json

Key Testing Files in the Repository

File	Role	Location
quick_validate.py	Front-matter validator used for every skill	skills/skill-creator/scripts/quick_validate.py
with_server.py	Helper to start a local server and run Playwright-based tests	skills/webapp-testing/scripts/with_server.py
evaluation.py	Generic evaluation harness for functional testing	skills/mcp-builder/scripts/evaluation.py
SKILL.md (webapp-testing)	Documentation of the web-app testing skill, includes testing instructions	skills/webapp-testing/SKILL.md
SKILL.md (skill-creator)	Specification of how a skill should be structured, including testing recommendations	skills/skill-creator/SKILL.md
agent-skills-spec.md	The formal Agent Skills specification that the validator enforces	spec/agent-skills-spec.md

Summary

• Front-matter validation ensures your SKILL.md metadata conforms to the Agent Skills specification using quick_validate.py.
• Unit-style testing executes helper scripts to catch runtime errors before deployment.
• Functional integration testing uses the evaluation.py harness to verify skill outputs match expected schemas.
• Manual UI verification provides a final visual check for skills generating web artifacts or images.
• Automated pipelines can combine all stages using bash scripts to prevent broken skills from reaching production.

Frequently Asked Questions

What happens if my skill fails front-matter validation?

If your skill fails front-matter validation, the quick_validate.py script will output specific error messages indicating which required fields are missing or malformed. For example, it will report unexpected keys, naming convention violations (such as names exceeding 64 characters or not using kebab-case), or description length issues. You must fix these errors before the skill can load in Claude's environment.

Can I test skills that require external services or APIs?

Yes, you can test skills that require external services using the unit-style script testing approach. The with_server.py helper script is designed to manage external dependencies by starting local servers (such as npm development servers) before executing your test scripts. For external APIs, ensure your test scripts mock or safely interact with these services to prevent accidental data modification during testing.

How does the functional integration harness simulate Claude requests?

The functional integration harness in skills/mcp-builder/scripts/evaluation.py simulates Claude requests by loading your skill and invoking it with predefined questions from a JSON evaluation file. It mimics the production runtime environment by processing the skill's output and comparing it against expected patterns, such as specific JSON schemas, image formats, or PDF structures. This ensures the skill's contract with Claude remains valid before deployment.

Is manual UI verification necessary for all skills?

No, manual UI verification is only necessary for skills that generate visual artifacts such as web applications, images, or documents requiring subjective visual assessment. Skills that process data or return text responses can rely solely on automated validation stages. The manual verification stage uses Playwright-based workflows documented in skills/webapp-testing/SKILL.md to inspect rendered output in a controlled browser environment.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}