PM Skills A/B Test Analysis: End-to-End Experiment Validation and Decision-Making

PM Skills provides a comprehensive A/B test analysis capability that guides product managers through experiment validation, statistical testing, and actionable decision-making via the /analyze-test command.

The phuryn/pm-skills repository offers a dedicated A/B test analysis skill designed to streamline experimentation workflows for product managers. This capability handles everything from raw data ingestion to final recommendations, ensuring statistical rigor throughout the experiment lifecycle according to the source code in pm-data-analytics/skills/ab-test-analysis/SKILL.md.

Core A/B Test Analysis Workflow

The A/B test analysis skill implements a seven-step methodology exposed through the /analyze-test command defined in pm-data-analytics/commands/analyze-test.md.

Input Handling and Data Ingestion

PM Skills accepts multiple input formats including conversion-rate summaries, raw CSV or Excel event logs, screenshots from platforms like Optimizely or LaunchDarkly, and plain-text descriptions. The skill's Instructions and command Invocation section define these input handlers to accommodate various data sources without requiring manual preprocessing.

Test Design Validation

Before running calculations, the system validates experimental design using power analysis and sample-size adequacy checks. It applies the formula n = (Z²α/2 × 2 × p × (1‑p)) / MDE² to flag under-powered tests and verifies duration spans at least 1–2 business cycles while checking for randomization quality and sample-ratio mismatch issues.

Statistical Calculations

When raw data is provided, the skill generates Python scripts using scipy.stats to compute conversion rates, relative lift, two-tailed z-tests or chi-square tests for p-values, and 95% confidence intervals. This ensures reproducible statistical analysis with practical significance thresholds that meet product decision standards.

Guardrail Assessment

The analysis evaluates secondary metrics such as revenue, engagement, and performance to detect hidden regressions that might not appear in primary conversion metrics. This explicit guardrail section in the skill description prevents shipping changes that negatively impact other business-critical indicators.

Decision Matrix and Recommendations

Results translate into one of four product actions: Ship, Investigate, Extend, or Stop. A markdown table in pm-data-analytics/skills/ab-test-analysis/SKILL.md maps statistical outcomes to specific business recommendations, bridging the gap between p-values and actionable product decisions.

Reporting and Follow-Up

The skill generates a markdown summary containing the hypothesis, duration, sample sizes, metric tables, statistical results, and concrete next steps. It also offers to design follow-up experiments, run segment-level analyses, or produce monitoring SQL under Step 5: Offer Next Steps.

Input Methods and Data Formats

The /analyze-test command supports flexible data ingestion across multiple formats:

• Summary statistics: Direct conversion rates and sample sizes provided as text
• Raw data files: CSV or Excel event logs for automated processing
• Visual inputs: Screenshots from experimentation platforms
• Text descriptions: Plain-language experiment details for manual entry

When processing raw data files, PM Skills automatically generates Python analysis scripts to ensure consistent statistical methodology.

Statistical Validation and Calculations

Statistical rigor is implemented through automated calculations defined in the skill documentation. The system performs two-tailed z-tests or chi-square tests to determine statistical significance while calculating confidence intervals for precise effect size estimation.

The power analysis formula ensures experiments meet minimum sample size requirements before drawing conclusions. When processing raw data, PM Skills dynamically generates Python code leveraging scipy.stats to ensure fully reproducible analysis workflows.

Decision Framework and Reporting

The reporting template in pm-data-analytics/skills/ab-test-analysis/SKILL.md structures output to include:

1. Experiment hypothesis and duration
2. Sample size calculations and statistical power verification
3. Primary metric results with confidence intervals
4. Guardrail metric status and secondary impact assessment
5. Clear recommendation (Ship/Investigate/Extend/Stop)
6. Actionable next steps for product development

Summary

• PM Skills provides an end-to-end A/B test analysis workflow through the /analyze-test command defined in pm-data-analytics/commands/analyze-test.md
• The skill validates experimental design using power analysis formulas before calculating statistical results
• Automated computations use scipy.stats for reproducible z-tests and chi-square tests when handling raw data
• Four-state decision matrix (Ship, Investigate, Extend, Stop) translates statistics into concrete product actions
• Comprehensive reporting includes guardrail metrics and follow-up experiment design capabilities

Frequently Asked Questions

What input formats does PM Skills accept for A/B test analysis?

PM Skills accepts conversion-rate summaries, raw CSV or Excel files, screenshots from platforms like Optimizely or LaunchDarkly, and plain-text descriptions. When raw data files are provided, the system automatically generates Python analysis scripts using scipy.stats to ensure statistical accuracy.

How does PM Skills calculate statistical significance?

The system computes two-tailed z-tests or chi-square tests using scipy.stats libraries to determine p-values and calculates 95% confidence intervals for effect sizes. It also validates sample size adequacy using the power analysis formula n = (Z²α/2 × 2 × p × (1‑p)) / MDE² to ensure experiments are properly powered before drawing conclusions.

What decisions can PM Skills recommend after analysis?

Based on statistical results and business context, the skill recommends one of four actions: Ship (deploy the variant), Investigate (examine anomalies or specific segments), Extend (run longer to reach statistical significance), or Stop (end the experiment without deploying the variant).

Does PM Skills check for negative impacts beyond the primary metric?

Yes, the skill includes a guardrail assessment that evaluates secondary metrics such as revenue, engagement, and performance to ensure the variant does not introduce regressions in other critical business areas. This prevents shipping changes that improve conversion rates but harm overall business health.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}