How to Design Effective Experiments for Testing Product Assumptions

TLDR: The phuryn/pm-skills repository provides a systematic, markdown-based skill that guides product managers through a five-step framework to design low-effort experiments, enabling rapid validation of assumptions using real behavioral metrics rather than opinions.

Designing effective experiments for testing product assumptions is essential for de-risking product decisions without wasting engineering resources. The PM Skills Marketplace open-source repository ships a dedicated brainstorm-experiments-existing skill that implements a structured framework for experiment design. This guide walks through the exact methodology defined in the source code, including practical prompts you can deploy immediately with any AI assistant.

The 5-Step Experiment Design Framework

The core logic resides in pm-product-discovery/skills/brainstorm-experiments-existing/SKILL.md, which structures the process into five distinct phases.

Step 1: Clarify the Idea and Assumptions

Before selecting experiment types, capture the feature idea, target user, and specific hypotheses requiring validation. The skill consumes this context in its Context section to tailor subsequent recommendations to your specific scenario.

Step 2: Choose Experiment Types

Select from the predefined experiment toolbox:

• First-click prototypes – Test initial user engagement with a mocked interface.
• Fake-door tests – Measure intent by presenting a non-functional feature option.
• Technical spikes – Validate technical feasibility with time-boxed exploration.
• A/B tests – Compare variants with controlled exposure.
• Wizard-of-Oz experiments – Simulate full functionality while humans handle the backend manually.
• Survey-based validation – Gather quantitative data on user preferences.

Step 3: Apply Key Principles

The skill enforces three non-negotiable principles:

• Measure real behavior, not opinion.
• Mitigate risk (e.g., limit exposure in A/B tests).
• Maximize learning per unit of effort.

Step 4: Flesh Out Each Experiment

For every assumption, define four specific fields:

• Assumption – What you believe is true.
• Experiment – Concrete execution steps (e.g., "launch a fake-door button for Feature X").
• Metric – Measurable user action (e.g., click-through rate).
• Success Threshold – The numeric value that confirms the assumption (e.g., "≥ 12% CTR").

Step 5: Present Results

Render a markdown table or structured document so the team can review and iterate. The skill recommends a clear table layout for easy scanning.

Practical Implementation: Using the brainstorm-experiments-existing Skill

Because the skill is pure markdown, it works with any AI assistant that reads SKILL.md files (Claude, Gemini, OpenCode, etc.). No code execution is required—only the structured guidance it contains.

Example 1: Simple One-Assumption Experiment

Trigger the skill by describing your assumption and requesting the specific output format:

Design experiments for the following assumption:

**Assumption:** Users will click "Try Free" on the landing page if we show a "30-day trial" badge.

Please output a markdown table with columns: Assumption, Experiment, Metric, Success Threshold.

Expected Output:

Assumption	Experiment	Metric	Success Threshold
Users will click "Try Free" when a "30-day trial" badge is displayed.	Add a temporary banner with the badge on the homepage (A/B test).	Click-through rate on "Try Free".	≥ 12% CTR (baseline 8%).

Example 2: Full Discovery Chain with /discover

For a complete discovery workflow, use the /discover command defined in pm-product-discovery/commands/discover.md. This orchestrates a four-step chain:

/discover
AI-powered meeting summarizer for remote teams

The command executes sequentially:

1. brainstorm-ideas-existing – Generate ideas for the summarizer.
2. identify-assumptions-existing – Surface risky assumptions (e.g., "users will tolerate 2-minute latency").
3. prioritize-assumptions – Rank assumptions on the Impact × Risk matrix.
4. brainstorm-experiments-existing – Produce the experiment table for each high-impact/high-risk assumption.

Repository Architecture and Key Files

Understanding the file structure helps you customize the framework or integrate it into custom workflows.

• pm-product-discovery/skills/brainstorm-experiments-existing/SKILL.md – Contains the core experiment-design logic, step-by-step guide, and output format specifications.
• pm-product-discovery/skills/prioritize-assumptions/SKILL.md – Provides the Impact × Risk matrix used to determine which assumptions need experiments first.
• pm-product-discovery/commands/discover.md – Defines the command that chains the full discovery workflow from ideation to experiment design.
• CLAUDE.md – Definitive agent guidance specifying how AI systems load and invoke skills.

Summary

• The brainstorm-experiments-existing skill in phuryn/pm-skills provides a five-step framework for designing product experiments.
• Each experiment requires four defined fields: Assumption, Experiment, Metric, and Success Threshold.
• The toolbox includes six experiment types: first-click prototypes, fake-door tests, technical spikes, A/B tests, Wizard-of-Oz, and survey-based validation.
• Use the /discover command to run the full discovery chain: brainstorming ideas, identifying assumptions, prioritizing them, and designing experiments.
• All skills are pure markdown, making them compatible with Claude, Gemini, OpenCode, and other AI assistants without requiring code execution.

Frequently Asked Questions

What is the difference between a fake-door test and a Wizard-of-Oz experiment?

A fake-door test measures intent by presenting a non-functional feature (like a button that shows "coming soon") to gauge click-through rates before building. A Wizard-of-Oz experiment appears fully functional to users, but a human manually performs the backend work behind the scenes. Both are supported experiment types in the brainstorm-experiments-existing skill according to the repository source code.

How do I prioritize which assumptions to test first?

Use the prioritize-assumptions skill, which plots assumptions on an Impact × Risk matrix. High-impact, high-risk assumptions should be validated first through experiments designed by the brainstorm-experiments-existing skill. This prioritization logic is defined in pm-product-discovery/skills/prioritize-assumptions/SKILL.md.

Can I use these experiment design skills without Claude?

Yes. The skills are pure markdown files that work with any AI assistant capable of reading SKILL.md instructions, including Gemini, OpenCode, and custom GPT implementations. The repository's CLAUDE.md file provides the standard for agent compatibility but does not restrict usage to Claude alone.

Where is the success threshold defined in the experiment output?

The success threshold is the fourth column in the markdown table generated by the skill. According to SKILL.md, this numeric value (e.g., "≥ 12% CTR") determines whether the assumption is validated or invalidated based on the measured metric, allowing teams to make objective go/no-go decisions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}