# How to Analyze A/B Test Results with Statistical Significance: A Product Manager's Guide

> Learn to analyze A/B test results with statistical significance. Validate setup, calculate conversion rates and p-values, and use a decision framework with the pm-skills repository.

- Repository: [Pawel Huryn/pm-skills](https://github.com/phuryn/pm-skills)
- Tags: how-to-guide
- Published: 2026-06-11

---

**You analyze A/B test results with statistical significance by validating experimental setup, calculating conversion rates and p-values using z-tests or chi-square tests, checking guardrail metrics, and mapping outcomes to a five-bucket decision framework—all automated through the pm-skills repository's structured workflow.**

The **pm-skills** repository provides product managers with a reproducible, code-driven approach to **analyze A/B test results with statistical significance**, eliminating guesswork from experiment interpretation. Based on the `A/B Test Analysis` skill in the data-analytics module, this workflow ensures you only ship features backed by rigorous statistical evidence.

## The A/B Test Analysis Workflow

The skill implements a six-stage pipeline defined in [`pm-data-analytics/skills/ab-test-analysis/SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/pm-data-analytics/skills/ab-test-analysis/SKILL.md) that transforms raw experiment data into actionable product decisions.

### Context Capture

Before running numbers, the skill records the experiment’s business context: your hypothesis, variant names, primary metric, test duration, and traffic split【SKILL.md#L10-L13】. This documentation ensures subsequent calculations align with the original intent and prevents cherry-picking metrics post-hoc.

### Setup Validation

The workflow validates whether your test was properly powered to detect meaningful differences:

- **Sample Size Check**: Uses the classic formula  
  \[
  n = \frac{Z_{\alpha/2}^2 \times 2 \times p(1-p)}{MDE^2}
  \]  
  to flag under-powered tests (those with less than 80% power)【SKILL.md#L26-L29】.

- **Duration Verification**: Confirms the test spans 1–2 full business cycles to account for weekly seasonality.

- **Randomization Audit**: Inspects for sample-ratio mismatch (SRM) to ensure traffic split integrity.

- **Novelty Effect Detection**: Flags early-behavior spikes that might skew results before users adapt to the new experience.

### Statistical Calculations

For the primary metric, the skill computes【SKILL.md#L33-L38】:

- **Conversion rates** for control and variant groups
- **Relative lift** percentage between groups
- **Two-tailed p-value** using z-tests (for continuous metrics) or chi-square tests (for categorical data)
- **95% confidence intervals** to quantify the range of plausible effect sizes

### Guardrail Metrics Check

Even when primary metrics improve, secondary health metrics must hold steady. The skill cross-references guardrail metrics—such as revenue per user, page load latency, or support ticket volume—to prevent "false wins" where user experience degrades in untracked ways【SKILL.md#L43-L46】.

### Decision Framework

Based on both statistical and practical significance, the skill maps results to one of five decision buckets【SKILL.md#L49-L55】:

1. **Ship** – Statistically significant positive lift with no guardrail degradation.
2. **Investigate** – Significant result but with concerning guardrail trends or implementation bugs.
3. **Extend** – Inconclusive results but promising trends; collect more data.
4. **Stop** – Negative significant result or severe guardrail violation.
5. **Don’t Ship** – Flat results with no meaningful lift.

### Automated Reporting

Finally, the skill generates a markdown summary including the hypothesis, key metrics table, and next-step guidance【SKILL.md#L57-L71】, creating an audit trail for stakeholders.

## Running the Analysis

The `pm-data-analytics` CLI automates the entire workflow. When you point it at raw CSV or Excel data, it validates power assumptions, runs statistical tests, and prints the markdown summary.

### CLI Command

Execute the built-in `analyze-test` command from the repository root:

```bash
pm-data-analytics analyze-test \
  --file experiment.csv \
  --hypothesis "New signup flow increases conversion" \
  --metric conversion \
  --duration 14 \
  --traffic-split 50

```

This parses `experiment.csv`, performs the calculations, and outputs the decision framework analysis.

### Standalone Python Implementation

For custom pipelines or Jupyter notebooks, the skill auto-generates reproducible Python scripts using **pandas** and **scipy**. Here is the core statistical logic:

```python
import pandas as pd
import scipy.stats as stats

# Load raw data (one row per user)

df = pd.read_csv("experiment.csv")
control = df[df.variant == "control"]
variant = df[df.variant == "variant"]

# Conversion rates

p_control = control.conversion.mean()
p_variant = variant.conversion.mean()

# Relative lift calculation

lift = (p_variant - p_control) / p_control * 100

# Two-tailed z-test for proportions

n_control = len(control)
n_variant = len(variant)
p_pool = (control.conversion.sum() + variant.conversion.sum()) / (n_control + n_variant)
se = (p_pool * (1 - p_pool) * (1/n_control + 1/n_variant)) ** 0.5
z = (p_variant - p_control) / se
p_value = 2 * (1 - stats.norm.cdf(abs(z)))

# 95% confidence interval for the difference

ci_low = (p_variant - p_control) - 1.96 * se
ci_high = (p_variant - p_control) + 1.96 * se

print(f"Control CR: {p_control:.2%}")
print(f"Variant CR: {p_variant:.2%}")
print(f"Lift: {lift:.2f}%")
print(f"p-value: {p_value:.4f}")
print(f"95% CI: [{ci_low:.2%}, {ci_high:.2%}]")
print("Significant?", "Yes" if p_value < 0.05 else "No")

```

This script yields identical results to the CLI tool, making it easy to embed into CI pipelines or analytic notebooks.

## Interpreting the Results

The skill produces structured markdown reports that align team understanding. A typical output looks like this:

```markdown

## A/B Test Results: New Signup Flow

**Hypothesis:** New signup flow increases conversion  
**Duration:** 14 days | **Sample:** 12,500 control / 12,450 variant  

| Metric      | Control | Variant | Lift | p-value | Significant? |
|------------|---------|--------|------|---------|--------------|
| Conversion | 2.34%   | 2.78%  | +18.8% | 0.0021 | Yes |

**Recommendation:** Ship it – roll out to 100%  
**Reasoning:** Statistically significant lift with no guard-rail degradation.  
**Next steps:** Deploy feature flag, monitor real-time metrics.

```

## Summary

- **Analyze A/B test results with statistical significance** using the six-stage workflow in [`pm-data-analytics/skills/ab-test-analysis/SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/pm-data-analytics/skills/ab-test-analysis/SKILL.md): context capture, setup validation, statistical calculation, guardrail checks, decision mapping, and automated reporting.
- Validate experimental power using the sample size formula before interpreting results to avoid false negatives.
- Use **z-tests** for conversion rate analysis and **chi-square tests** for categorical outcomes, implemented in the auto-generated Python scripts.
- Always check guardrail metrics to prevent shipping features that improve primary metrics while degrading overall product health.
- Map outcomes to the five-bucket framework (Ship, Investigate, Extend, Stop, Don’t Ship) to ensure consistent decision-making across your organization.

## Frequently Asked Questions

### What statistical test does the pm-skills repository use for A/B test analysis?

The repository primarily uses **two-tailed z-tests** for comparing conversion rates between control and variant groups, and **chi-square tests** for categorical data analysis. These methods calculate the p-value and 95% confidence intervals while accounting for pooled standard errors between groups, as implemented in the Python scripts generated by the `analyze-test` command.

### How does the workflow prevent false positives in A/B testing?

The workflow prevents false positives through three safeguards: it requires **setup validation** to ensure adequate sample size (80% power minimum), implements **guardrail metric checks** to catch secondary metric degradation, and mandates **business cycle duration** (1-2 weeks minimum) to avoid novelty effects. These checks are enforced before any significance calculations are interpreted【SKILL.md#L26-L46】.

### Can I use this analysis workflow with my existing experiment data?

Yes. The skill accepts standard CSV or Excel formats with one row per user containing variant assignment and conversion flags. Running `pm-data-analytics analyze-test --file your-data.csv` automatically parses the data, validates the experimental setup, runs the statistical calculations, and outputs the markdown decision framework without requiring manual data transformation.

### What does it mean if my test results are statistically significant but the recommendation is "Investigate"?

This occurs when the primary metric shows a significant p-value (typically < 0.05) but guardrail metrics reveal concerning trends—such as revenue drops or latency increases—or the sample ratio mismatch (SRM) check detects randomization issues. According to the decision matrix in [`SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/SKILL.md), significant lift with broken guardrails triggers the "Investigate" bucket rather than "Ship" to prevent deploying harmful optimizations【SKILL.md#L43-L55】.