# How to Analyze A/B Test Results for Statistical Significance: A Complete Guide

> Learn how to analyze A/B test results for statistical significance. This guide covers p-values, confidence intervals, and making product decisions with the five-bucket framework.

- Repository: [Pawel Huryn/pm-skills](https://github.com/phuryn/pm-skills)
- Tags: how-to-guide
- Published: 2026-06-13

---

**Analyzing A/B test results for statistical significance requires validating your experimental setup, calculating p-values and confidence intervals using z-tests or chi-square tests, and mapping outcomes to clear product decisions using the five-bucket framework defined in the pm-skills repository.**

The **pm-skills** repository provides a structured, data-driven workflow for turning raw experiment data into actionable product decisions. Located in [`pm-data-analytics/skills/ab-test-analysis/SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/pm-data-analytics/skills/ab-test-analysis/SKILL.md), this resource outlines a complete pipeline that ensures product managers avoid common pitfalls like under-powered tests or false positives while analyzing A/B test results for statistical significance.

## Prerequisites and Context Capture

Before running calculations, the skill captures essential business context to ensure the analysis aligns with strategic goals. According to the source code in [`SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/SKILL.md) (lines 10-13), you must document:

- **Hypothesis**: The expected change and business impact
- **Variant**: Description of the treatment being tested
- **Primary metric**: The single success criterion (e.g., conversion rate)
- **Test duration**: Length of the experiment in days
- **Traffic split**: Percentage allocation between control and variant

This context prevents "cherry-picking" metrics post-hoc and ensures the statistical test remains valid.

## Validating Experimental Setup

Rigorous A/B test analysis begins with validation checks to ensure the experiment was properly powered and executed correctly.

### Sample Size and Statistical Power

The skill validates whether your test had adequate statistical power using the classic sample size formula referenced in [`SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/SKILL.md) (lines 26-29):

```

n = (Z² × 2 × p(1-p)) / MDE²

```

Where:
- **Z** = Z-score for your confidence level (1.96 for 95% confidence)
- **p** = Baseline conversion rate
- **MDE** = Minimum Detectable Effect (the smallest lift you want to detect)

The workflow flags under-powered tests when power falls below **80%**, as these results lack the sensitivity to detect meaningful differences reliably.

### Duration and Randomization Checks

Beyond sample size, the skill verifies three critical execution elements:

- **Duration**: Spanning at least 1–2 full business cycles to account for day-of-week effects
- **Randomization**: Inspecting for **Sample Ratio Mismatch (SRM)**, where the actual traffic split deviates significantly from the intended 50/50 or other configured ratio
- **Novelty effects**: Checking for primacy or novelty spikes where early user behavior skews overall results

## Calculating Statistical Significance

Once validation passes, the skill computes core statistical metrics using either z-tests for proportions or chi-square tests for categorical data.

### Core Metrics and Lift Calculation

The analysis calculates:
- **Conversion rates** for both control and variant groups
- **Relative lift**: `(variant_rate - control_rate) / control_rate × 100`

### P-Values and Confidence Intervals

For statistical significance testing, the implementation in [`SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/SKILL.md) (lines 33-38) uses a **two-tailed z-test** to calculate:

- **P-value**: The probability that the observed difference occurred by chance
- **95% confidence interval**: The range where the true difference likely falls

A result is considered statistically significant when **p < 0.05** and the confidence interval does not include zero.

## Guardrail Metrics and Business Logic

Statistical significance alone does not guarantee a shipping decision. The skill performs a **guardrail check** (lines 43-46) to prevent "false wins" where the primary metric improves while secondary health metrics degrade.

Guardrail metrics typically include:
- Revenue per user
- Page load latency
- Feature adoption rates
- Error rates

If guardrail metrics decline significantly, the recommendation shifts from "Ship" to "Investigate" regardless of primary metric lift.

## Interpreting Results and Decision Making

Based on both statistical and practical significance, the skill maps outcomes to one of five decision buckets defined in [`SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/SKILL.md) (lines 49-55):

- **Ship**: Statistically significant positive lift with no guardrail degradation
- **Investigate**: Significant results with guardrail concerns or unexpected patterns
- **Extend**: Directional positive results lacking statistical power (under-powered test)
- **Stop**: Negative statistically significant impact
- **Don't ship**: Inconclusive results with no meaningful lift

## Implementation: Running the Analysis

The repository provides two methods for executing the analysis workflow: a CLI command for quick execution and a standalone Python script for customization.

### Using the CLI Command

The `analyze-test` command defined in [`pm-data-analytics/commands/analyze-test.md`](https://github.com/phuryn/pm-skills/blob/main/pm-data-analytics/commands/analyze-test.md) provides a streamlined interface:

```bash
pm-data-analytics analyze-test \
  --file experiment.csv \
  --hypothesis "New signup flow increases conversion" \
  --metric conversion \
  --duration 14 \
  --traffic-split 50

```

This command parses your CSV, validates power requirements, computes significance, and outputs a markdown report following the skill's templates.

### Standalone Python Script

For deeper customization or integration into data pipelines, the skill generates Python code that implements the core statistical logic:

```python
import pandas as pd
import scipy.stats as stats

# Load raw data (one row per user)

df = pd.read_csv("experiment.csv")
control = df[df.variant == "control"]
variant = df[df.variant == "variant"]

# Conversion rates

p_control = control.conversion.mean()
p_variant = variant.conversion.mean()

# Lift calculation

lift = (p_variant - p_control) / p_control * 100

# Two-tailed z-test for proportions

n_control = len(control)
n_variant = len(variant)
p_pool = (control.conversion.sum() + variant.conversion.sum()) / (n_control + n_variant)
se = (p_pool * (1 - p_pool) * (1/n_control + 1/n_variant)) ** 0.5
z = (p_variant - p_control) / se
p_value = 2 * (1 - stats.norm.cdf(abs(z)))

# 95% confidence interval

ci_low = (p_variant - p_control) - 1.96 * se
ci_high = (p_variant - p_control) + 1.96 * se

print(f"Control CR: {p_control:.2%}")
print(f"Variant CR: {p_variant:.2%}")
print(f"Lift: {lift:.2f}%")
print(f"p-value: {p_value:.4f}")
print(f"95% CI: [{ci_low:.2%}, {ci_high:.2%}]")
print("Significant?", "Yes" if p_value < 0.05 else "No")

```

### Output Format

The skill produces a standardized markdown report template (lines 57-71) that includes:

```markdown

## A/B Test Results: New Signup Flow

**Hypothesis:** New signup flow increases conversion  
**Duration:** 14 days | **Sample:** 12,500 control / 12,450 variant  

| Metric | Control | Variant | Lift | p-value | Significant? |
|--------|---------|---------|------|---------|--------------|
| Conversion | 2.34% | 2.78% | +18.8% | 0.0021 | Yes |

**Recommendation:** Ship it – roll out to 100%  
**Reasoning:** Statistically significant lift with no guardrail degradation.  
**Next steps:** Deploy feature flag, monitor real-time metrics.

```

## Summary

- **Validate first**: Check sample size power, duration, and randomization (SRM) before analyzing results
- **Use proper formulas**: Calculate p-values using two-tailed z-tests and report 95% confidence intervals
- **Check guardrails**: Always verify secondary metrics haven't degraded before shipping
- **Follow the decision matrix**: Map results to the five buckets (Ship, Investigate, Extend, Stop, Don't ship)
- **Automate reporting**: Use the `pm-data-analytics analyze-test` command or generated Python scripts to ensure reproducible analysis

## Frequently Asked Questions

### What sample size do I need for an A/B test?

Calculate sample size using the formula `n = (Z² × 2 × p(1-p)) / MDE²` where Z is 1.96 for 95% confidence, p is your baseline conversion rate, and MDE is your minimum detectable effect. According to the pm-skills implementation, you need sufficient sample size to achieve at least 80% statistical power to reliably detect your target effect size.

### How do I calculate statistical significance for conversion rates?

Use a two-tailed z-test for proportions by calculating the pooled standard error, computing the Z-score from the difference in conversion rates, and deriving the p-value from the standard normal distribution. The pm-skills repository provides this implementation in [`SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/SKILL.md) (lines 33-38), which also calculates 95% confidence intervals to determine if the difference is practically meaningful.

### What is a guardrail metric in A/B testing?

Guardrail metrics are secondary health indicators (such as revenue per user, latency, or error rates) that must remain stable even when your primary metric improves. The pm-skills workflow explicitly checks these in [`SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/SKILL.md) (lines 43-46) to prevent shipping features that optimize one metric while degrading overall system health or user experience.

### When should I ship an A/B test variant?

Ship only when you observe statistically significant positive lift (p < 0.05), the confidence interval excludes zero, guardrail metrics show no degradation, and the effect size meets your minimum detectable threshold. The pm-skills repository classifies this as the "Ship" decision bucket, distinguishing it from "Investigate" (concerning patterns), "Extend" (under-powered), or "Stop" (negative impact).