How to Test a Custom Skill Before Submission in the pm-skills Repository

Run python validate_plugins.py to validate metadata and schema, then execute pm test-scenarios <skill-name> to verify runtime behavior against expected outputs before submitting to the marketplace.

Testing a custom skill before submission ensures your contribution meets the pm-skills repository standards and functions correctly in the PM-Toolkit ecosystem. The repository provides automated validation tools and scenario testing commands that verify both structural integrity and runtime behavior. Following the official testing workflow prevents broken submissions and guarantees your skill behaves as described under realistic conditions.

Prerequisites and File Structure

Before running tests, organize your skill files according to the repository layout and register the skill in the plugin manifest.

Create the Skill Directory

Place your skill under pm-toolkit/skills/<your-skill>/ containing a SKILL.md file. This directory houses all files specific to your skill, including documentation and metadata.

Register in the Plugin Manifest

Add an entry to pm-toolkit/.claude-plugin/plugin.json specifying the skill name, type, and path to your SKILL.md:

{
  "name": "my-awesome-skill",
  "type": "skill",
  "path": "pm-toolkit/skills/my-awesome-skill/SKILL.md"
}

The CLI uses this manifest to discover and load available skills.

Step-by-Step Testing Workflow

Step 1: Validate Plugin Metadata

Run the repository validator to check syntactic correctness:

python validate_plugins.py

The script, located at the repository root, parses every SKILL.md and the plugin manifest, reporting missing fields or malformed JSON. A clean run indicates the skill passes the syntactic checks required for marketplace submission.

Step 2: Create Scenario Tests

Add a Markdown file under pm-execution/skills/test-scenarios/<your-skill>.md containing the test specification:

# Test: my-awesome-skill

**Prompt**
Write a 2-sentence brief for the idea: "AI-powered habit tracker".

**Expected Output**
- Must contain the phrase "habit tracker".
- Must be ≤ 2 sentences.

This file defines the exact user input and the constraints the skill output must satisfy.

Step 3: Execute Scenario Tests

Run the scenario testing harness from the repository root:

pm test-scenarios my-awesome-skill

As documented in pm-execution/commands/test-scenarios.md, this command feeds the prompt to the skill, captures the response, and diff-compares it against the expected output constraints.

Step 4: Manual CLI Verification

After automated tests pass, invoke the skill directly to inspect output formatting and edge cases:

pm run my-awesome-skill --prompt "Write a brief for: AI-powered habit tracker"

Test with empty inputs, long strings, and special characters to ensure robustness.

Complete Code Examples

Minimal SKILL.md Template

Create pm-toolkit/skills/my-awesome-skill/SKILL.md:

# Skill: my-awesome-skill

**Description**
Generate a concise project brief from a short product idea.

**Parameters**
- `idea` (string): one-sentence product concept.

**Example Prompt**

Write a 2-sentence brief for the idea: "AI-powered habit tracker".

**Result**
A markdown paragraph summarising the product vision.

See a real example in the repository: pm-toolkit/skills/review-resume/SKILL.md.

Updating the Plugin Manifest

Edit pm-toolkit/.claude-plugin/plugin.json to include your skill:

{
  "plugins": [
    {
      "name": "my-awesome-skill",
      "type": "skill",
      "path": "pm-toolkit/skills/my-awesome-skill/SKILL.md"
    }
  ]
}

Running the Validator

Execute from the repository root:

python validate_plugins.py

# → Output: 0 errors, 0 warnings

The validator checks that all registered skills have valid SKILL.md files and that the JSON syntax in plugin.json is correct.

Adding a Scenario Test

Create pm-execution/skills/test-scenarios/my-awesome-skill.md:

# Test: my-awesome-skill

**Prompt**
Write a 2-sentence brief for the idea: "AI-powered habit tracker".

**Expected Output**
- Must contain the phrase "habit tracker".
- Must be ≤ 2 sentences.

Executing the Test

Run the specific scenario test:

pm test-scenarios my-awesome-skill

# → PASSED: output matches expected constraints

Manual CLI Invocation

Perform final sanity checks:

pm run my-awesome-skill --prompt "Write a brief for: AI-powered habit tracker"

Summary

• Validate first: Run python validate_plugins.py to catch syntax errors and missing metadata before functional testing.
• Test scenarios: Place scenario files in pm-execution/skills/test-scenarios/ and run pm test-scenarios <skill-name> to verify behavior against expected outputs.
• Register properly: Ensure your skill is listed in pm-toolkit/.claude-plugin/plugin.json with the correct path to SKILL.md.
• Manual verification: Use pm run <skill-name> to test edge cases and formatting that automated tests might miss.
• Iterate: Repeat the validation and testing cycle until all checks pass and the output meets quality standards.

Frequently Asked Questions

What does validate_plugins.py check exactly?

The script validates that every skill registered in pm-toolkit/.claude-plugin/plugin.json has a corresponding SKILL.md file with the required header structure, valid JSON syntax in the manifest, and all required CLI files present. It reports missing fields, malformed paths, and schema violations that would prevent the skill from loading in the PM-Toolkit.

Where do I place scenario test files?

Create Markdown files in pm-execution/skills/test-scenarios/ named after your skill (e.g., my-awesome-skill.md). These files should contain a Prompt section with the exact input and an Expected Output section defining the constraints or content the skill must produce.

Can I test a skill without registering it in plugin.json?

No. The pm test-scenarios command and pm run command both rely on the plugin registry to locate and load skill definitions. You must add an entry to pm-toolkit/.claude-plugin/plugin.json pointing to your SKILL.md before the testing commands can recognize your skill.

How do I debug a failing scenario test?

Check the diff output from pm test-scenarios to see where the actual output diverged from the expected constraints. Verify that your SKILL.md logic handles the specific prompt correctly, and ensure the expected output constraints in your scenario file are not overly restrictive. Run pm run <skill-name> with the same prompt to inspect the raw output manually.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}