How to Test and Validate Custom Claude Skills: A Complete Guide

To test and validate custom Claude skills, verify the YAML metadata with scripts/package_skill.py, test the SKILL.md instructions across Claude.ai, Claude Code, and the Skills API, and validate bundled resources through direct execution and end-to-end evaluation harnesses.

Custom Claude skills in the ComposioHQ/awesome-claude-skills repository are plain-text packages that require rigorous validation across multiple environments. Because these skills follow a progressive-disclosure architecture—loading metadata first, then instructions, then bundled resources—testing must occur at each layer to ensure reliable performance. Understanding how to test and validate custom Claude skills ensures your automation works consistently whether users invoke it through the web UI, CLI, or API.

Understanding the Three-Layer Validation Architecture

Claude skills load in three distinct stages, and each requires specific validation according to the repository’s architecture defined in README.md (lines 100-104).

Metadata Validation (YAML Front-matter)

The SKILL.md file must begin with valid YAML front-matter containing name and description fields. Claude’s runtime parses this metadata before loading the skill body, and malformed front-matter prevents the skill from ever being considered by the agent. According to skill-creator/SKILL.md (lines 91-95), the scripts/package_skill.py utility validates this metadata automatically and fails with clear error messages if required fields are missing or malformed.

SKILL.md Body Instructions

Once metadata passes validation, Claude evaluates the instruction set to determine if the skill is relevant to the user’s prompt. The Skill Creator guide recommends a "test-then-iterate" loop after writing the initial instructions (skill-creator/SKILL.md, lines 203-210). You must verify that trigger conditions work correctly and that the instructions handle edge cases without ambiguity. This layer tests whether Claude correctly interprets when to activate your custom logic.

Bundled Resources (scripts/, references/, assets/)

The final layer includes executable scripts, reference documents, and binary assets that Claude loads on demand. These resources execute in the user’s environment, so their correctness is validated by direct invocation—for example, running python scripts/with_server.py --help for the Webapp Testing skill to verify the helper launches correctly (webapp-testing/SKILL.md, lines 14-28).

Testing Across the Three Claude Runtimes

According to CONTRIBUTING.md (lines 19-20), every skill must be tested across Claude.ai, Claude Code, and the API before submission. These environments have distinct characteristics:

• Claude.ai – The web UI where skills are automatically discovered from your account library. File system access is limited to the browser sandbox.
• Claude Code – A local CLI that loads skills from ~/.config/claude-code/skills/. This environment has full access to the local file system and network.
• Skills API – Programmatic access that sends a skills list with each request, requiring explicit skill IDs in the API call.

A skill that passes in one environment can still break in another due to differences in file-system layout, network transport, or authentication handling. The contribution checklist forces authors to verify portability across all three contexts.

Practical Testing Strategies and Code Examples

Step 1: Validate Metadata with package_skill.py

Run the built-in validator from your skill’s root directory to check YAML front-matter and directory layout:

# From the skill's root folder

scripts/package_skill.py .

# Output: "Validation passed – zip created: my-skill.zip"

This script, referenced in skill-creator/SKILL.md (lines 91-95), ensures your name and description fields are present and correctly formatted before you attempt runtime testing.

Step 2: Test Bundled Scripts Locally

For skills that include automation scripts, validate execution success and I/O correctness directly. The Webapp Testing skill demonstrates this pattern using a Playwright-based workflow:

# Start a local dev server (the helper manages the lifecycle)

python scripts/with_server.py --server "npm run dev" --port 5173 -- python my_test.py

Here, my_test.py contains only the Playwright logic, while with_server.py handles server startup and teardown (webapp-testing/SKILL.md, lines 39-50). Always test with --help first, then with realistic inputs to verify idempotence.

Step 3: Run End-to-End Evaluation Harnesses

For complex skills involving tool chains, create an evaluation harness following the MCP Builder pattern. Describe a realistic scenario in XML format:

<evaluation>
  <scenario>
    <prompt>Generate a weekly sales report from my Shopify store.</prompt>
    <expected_tool>shopify-automation</expected_tool>
  </scenario>
</evaluation>

Execute the validation with:

python scripts/evaluation.py evaluation.xml -o report.md

As documented in mcp-builder/reference/evaluation.md (lines 5-15 and 420-447), this harness spins up the MCP server if needed and validates that your skill calls the correct tool chain under realistic conditions.

Step 4: Validate via the Skills API

Test programmatic access using the Anthropic SDK to ensure your skill loads correctly when specified explicitly:

import anthropic

client = anthropic.Anthropic(api_key="YOUR_API_KEY")
response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    skills=["my-skill-id"],
    messages=[{"role": "user", "content": "Please run my custom skill"}],
)
print(response.content)

If the skill is correctly packaged and uploaded, Claude will load it automatically and produce the expected output path, confirming that the progressive-disclosure sequence (metadata → body → resources) executes without errors.

Summary

• Validate metadata first using scripts/package_skill.py to catch YAML front-matter errors before runtime testing.
• Test across three environments—Claude.ai, Claude Code, and the Skills API—to ensure portability according to CONTRIBUTING.md.
• Verify progressive disclosure by confirming skills load in the correct sequence: metadata validation → instruction loading → resource execution.
• Execute bundled scripts directly with helper tools like with_server.py to test lifecycle management and I/O correctness.
• Use evaluation harnesses for end-to-end validation of complex tool chains, following the MCP Builder XML pattern.

Frequently Asked Questions

What file structure is required for a valid Claude skill?

A valid custom Claude skill requires a SKILL.md file at the root containing YAML front-matter with name and description fields, followed by the instruction body. Optional directories include scripts/, references/, and assets/ for bundled resources. The scripts/package_skill.py validator checks this exact layout as defined in skill-creator/SKILL.md.

Why does my skill work in Claude.ai but fail in Claude Code?

Environment differences in file-system access, network transport, or authentication handling cause this discrepancy. Claude Code operates locally with full file system access, while Claude.ai runs in a browser sandbox. According to CONTRIBUTING.md, you must test in both environments plus the API to ensure the skill handles path resolution and permissions correctly across contexts.

How do I test skills that require external servers or browsers?

Follow the pattern in webapp-testing/SKILL.md using the with_server.py helper script. This utility starts your development server, waits for the port to become available, then executes your test script (such as Playwright browser automation), and finally cleans up the server process regardless of test success or failure.

Can I automate skill validation in CI/CD pipelines?

Yes. Use scripts/package_skill.py for automated metadata and structure validation, and integrate scripts/evaluation.py with XML scenario files for functional testing. These command-line tools return non-zero exit codes on failure, making them compatible with GitHub Actions, Jenkins, or other CI systems without requiring manual UI interaction.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}