# PM Skills A/B Test Analysis: End-to-End Experiment Validation and Decision-Making > Unlock powerful A/B test analysis with PM Skills. Validate experiments, perform statistical tests, and make data driven decisions using the analyze-test command. - Repository: [Pawel Huryn/pm-skills](https://github.com/phuryn/pm-skills) - Tags: how-to-guide - Published: 2026-06-20 --- **PM Skills provides a comprehensive A/B test analysis capability that guides product managers through experiment validation, statistical testing, and actionable decision-making via the `/analyze-test` command.** The `phuryn/pm-skills` repository offers a dedicated **A/B test analysis** skill designed to streamline experimentation workflows for product managers. This capability handles everything from raw data ingestion to final recommendations, ensuring statistical rigor throughout the experiment lifecycle according to the source code in [`pm-data-analytics/skills/ab-test-analysis/SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/pm-data-analytics/skills/ab-test-analysis/SKILL.md). ## Core A/B Test Analysis Workflow The A/B test analysis skill implements a seven-step methodology exposed through the `/analyze-test` command defined in [`pm-data-analytics/commands/analyze-test.md`](https://github.com/phuryn/pm-skills/blob/main/pm-data-analytics/commands/analyze-test.md). ### Input Handling and Data Ingestion PM Skills accepts multiple input formats including conversion-rate summaries, raw CSV or Excel event logs, screenshots from platforms like Optimizely or LaunchDarkly, and plain-text descriptions. The skill's *Instructions* and command *Invocation* section define these input handlers to accommodate various data sources without requiring manual preprocessing. ### Test Design Validation Before running calculations, the system validates experimental design using power analysis and sample-size adequacy checks. It applies the formula `n = (Z²α/2 × 2 × p × (1‑p)) / MDE²` to flag under-powered tests and verifies duration spans at least 1–2 business cycles while checking for randomization quality and sample-ratio mismatch issues. ### Statistical Calculations When raw data is provided, the skill generates Python scripts using `scipy.stats` to compute conversion rates, relative lift, two-tailed z-tests or chi-square tests for p-values, and 95% confidence intervals. This ensures reproducible statistical analysis with practical significance thresholds that meet product decision standards. ### Guardrail Assessment The analysis evaluates secondary metrics such as revenue, engagement, and performance to detect hidden regressions that might not appear in primary conversion metrics. This explicit guardrail section in the skill description prevents shipping changes that negatively impact other business-critical indicators. ### Decision Matrix and Recommendations Results translate into one of four product actions: **Ship**, **Investigate**, **Extend**, or **Stop**. A markdown table in [`pm-data-analytics/skills/ab-test-analysis/SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/pm-data-analytics/skills/ab-test-analysis/SKILL.md) maps statistical outcomes to specific business recommendations, bridging the gap between p-values and actionable product decisions. ### Reporting and Follow-Up The skill generates a markdown summary containing the hypothesis, duration, sample sizes, metric tables, statistical results, and concrete next steps. It also offers to design follow-up experiments, run segment-level analyses, or produce monitoring SQL under *Step 5: Offer Next Steps*. ## Input Methods and Data Formats The `/analyze-test` command supports flexible data ingestion across multiple formats: - **Summary statistics**: Direct conversion rates and sample sizes provided as text - **Raw data files**: CSV or Excel event logs for automated processing - **Visual inputs**: Screenshots from experimentation platforms - **Text descriptions**: Plain-language experiment details for manual entry When processing raw data files, PM Skills automatically generates Python analysis scripts to ensure consistent statistical methodology. ## Statistical Validation and Calculations Statistical rigor is implemented through automated calculations defined in the skill documentation. The system performs two-tailed z-tests or chi-square tests to determine statistical significance while calculating confidence intervals for precise effect size estimation. The power analysis formula ensures experiments meet minimum sample size requirements before drawing conclusions. When processing raw data, PM Skills dynamically generates Python code leveraging `scipy.stats` to ensure fully reproducible analysis workflows. ## Decision Framework and Reporting The reporting template in [`pm-data-analytics/skills/ab-test-analysis/SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/pm-data-analytics/skills/ab-test-analysis/SKILL.md) structures output to include: 1. Experiment hypothesis and duration 2. Sample size calculations and statistical power verification 3. Primary metric results with confidence intervals 4. Guardrail metric status and secondary impact assessment 5. Clear recommendation (Ship/Investigate/Extend/Stop) 6. Actionable next steps for product development ## Summary - PM Skills provides an end-to-end **A/B test analysis** workflow through the `/analyze-test` command defined in [`pm-data-analytics/commands/analyze-test.md`](https://github.com/phuryn/pm-skills/blob/main/pm-data-analytics/commands/analyze-test.md) - The skill validates experimental design using power analysis formulas before calculating statistical results - Automated computations use `scipy.stats` for reproducible z-tests and chi-square tests when handling raw data - Four-state decision matrix (Ship, Investigate, Extend, Stop) translates statistics into concrete product actions - Comprehensive reporting includes guardrail metrics and follow-up experiment design capabilities ## Frequently Asked Questions ### What input formats does PM Skills accept for A/B test analysis? PM Skills accepts conversion-rate summaries, raw CSV or Excel files, screenshots from platforms like Optimizely or LaunchDarkly, and plain-text descriptions. When raw data files are provided, the system automatically generates Python analysis scripts using `scipy.stats` to ensure statistical accuracy. ### How does PM Skills calculate statistical significance? The system computes two-tailed z-tests or chi-square tests using `scipy.stats` libraries to determine p-values and calculates 95% confidence intervals for effect sizes. It also validates sample size adequacy using the power analysis formula `n = (Z²α/2 × 2 × p × (1‑p)) / MDE²` to ensure experiments are properly powered before drawing conclusions. ### What decisions can PM Skills recommend after analysis? Based on statistical results and business context, the skill recommends one of four actions: **Ship** (deploy the variant), **Investigate** (examine anomalies or specific segments), **Extend** (run longer to reach statistical significance), or **Stop** (end the experiment without deploying the variant). ### Does PM Skills check for negative impacts beyond the primary metric? Yes, the skill includes a guardrail assessment that evaluates secondary metrics such as revenue, engagement, and performance to ensure the variant does not introduce regressions in other critical business areas. This prevents shipping changes that improve conversion rates but harm overall business health.