# A/B Test Analysis Skill Statistical Methods: Z-Tests, Power Analysis, and Effect Size

> Master A/B test analysis with statistical methods like Z-tests, power analysis, and effect size. Validate experiments and drive data-driven product decisions.

- Repository: [Pawel Huryn/pm-skills](https://github.com/phuryn/pm-skills)
- Tags: deep-dive
- Published: 2026-06-23

---

**The A/B test analysis skill in the phuryn/pm-skills repository employs a six-step statistical workflow combining power analysis, two-tailed Z-tests for proportions, confidence intervals, and relative lift calculations to validate experiments and drive product decisions.**

The A/B test analysis skill provides a structured framework for evaluating product experiments within the phuryn/pm-skills repository. This open-source implementation codifies standard statistical practices into an executable workflow that transforms raw conversion data into actionable recommendations through rigorous hypothesis testing and effect size estimation.

## Statistical Workflow Overview

The skill implements a sequential analytical pipeline defined in [`pm-data-analytics/skills/ab-test-analysis/SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/pm-data-analytics/skills/ab-test-analysis/SKILL.md). Each step applies specific statistical techniques to ensure experimental validity and practical relevance.

### Step 1: Validate Test Design with Power Analysis

Before analyzing results, the skill verifies whether the experiment was properly powered. It calculates the required sample size using the formula:

$n = (Z_{\alpha/2}^2 \times 2 \times p \times (1-p)) / MDE^2$

Where:
- $Z_{\alpha/2}$ represents the critical value for the desired confidence level (typically 1.96 for 95% confidence)
- $p$ denotes the pooled baseline conversion rate
- $MDE$ specifies the minimum detectable effect

According to the source code at lines 26-30, the skill flags under-powered tests that fall below 80% power or lack sufficient duration, preventing premature analysis of inconclusive data.

### Step 2: Calculate Conversion Rates

The skill computes **conversion rates** as simple proportions for both control and variant groups:

$p = \text{conversions} / \text{sample size}$

This calculation establishes the baseline metrics required for subsequent significance testing. The workflow processes raw CSV data containing user-level conversion events, aggregating them by variant assignment before proceeding to comparative analysis.

### Step 3: Test Statistical Significance

To determine whether observed differences exceed random variation, the skill implements a **two-tailed Z-test** for difference in proportions (equivalent to a chi-squared test for count data). As specified in [`SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/SKILL.md) lines 33-38, the analysis yields:

- **P-value**: Calculated as $2 \times (1 - \text{norm.cdf}(|z|))$ where $z = (p_v - p_c) / SE$
- **Standard Error**: Computed using the pooled proportion: $\sqrt{p_{pool}(1-p_{pool})(1/n_c + 1/n_v)}$
- **95% Confidence Interval**: Absolute difference $\pm 1.96 \times SE$

This approach follows the frequentist hypothesis testing framework, rejecting the null hypothesis when $p < 0.05$.

### Step 4: Assess Practical Significance

Statistical significance does not guarantee business value. The skill calculates **relative lift** to quantify practical impact:

$\text{Lift} = (\text{variant rate} - \text{control rate}) / \text{control rate} \times 100\%$

Lines 34-39 of [`SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/SKILL.md) indicate that this metric is compared against guard-rail thresholds to determine whether the observed effect size warrants implementation, regardless of statistical significance.

### Step 5: Segment and Guard-Rail Analysis

For deeper insights, the workflow supports optional segmentation of results by user attributes or secondary metrics. While no new statistical tests are mandated, the same Z-test or chi-squared methodology applies to subgroup analyses.

### Step 6: Decision Matrix

The final stage maps statistical outputs to concrete product actions. As defined in lines 49-56, the skill categorizes outcomes into four decisions:

- **Ship**: Statistically significant positive lift with adequate power
- **Extend**: Under-powered but promising trend requiring additional sample
- **Stop**: Significant negative impact or neutral results with sufficient power
- **Investigate**: Anomalous patterns or guard-rail violations requiring manual review

## Python Implementation Examples

The `/analyze-test` command orchestrates these calculations, generating Python code that leverages `pandas` and `scipy.stats`. Below is the complete analytical pipeline:

```python
import pandas as pd
import numpy as np
import scipy.stats as st

# Load experiment data

df = pd.read_csv("test_results.csv")

# Aggregate conversion counts by variant

summary = df.groupby("variant")["converted"].agg(["sum", "count"]).reset_index()
control = summary[summary["variant"] == "control"]
variant = summary[summary["variant"] == "variant"]

# Extract counts

c_conv, c_n = control["sum"].iloc[0], control["count"].iloc[0]
v_conv, v_n = variant["sum"].iloc[0], variant["count"].iloc[0]

# Calculate conversion rates

p_c = c_conv / c_n
p_v = v_conv / v_n

# Compute relative lift

lift = (p_v - p_c) / p_c * 100

# Two-tailed Z-test for proportions

pooled = (c_conv + v_conv) / (c_n + v_n)
se = np.sqrt(pooled * (1 - pooled) * (1/c_n + 1/v_n))
z = (p_v - p_c) / se
p_value = 2 * (1 - st.norm.cdf(abs(z)))

# 95% Confidence interval for difference

ci_margin = 1.96 * se
ci_low = (p_v - p_c) - ci_margin
ci_high = (p_v - p_c) + ci_margin

print(f"Control: {p_c:.2%} (n={c_n})")
print(f"Variant: {p_v:.2%} (n={v_n})")
print(f"Relative lift: {lift:.2f}%")
print(f"p-value: {p_value:.4f}")
print(f"95% CI: [{ci_low:.3%}, {ci_high:.3%}]")

```

### Power Analysis Verification

To validate the experimental design prior to analysis, the skill implements the power calculation formula referenced in the documentation:

```python
import math

# Parameters

alpha = 0.05
z_alpha = st.norm.ppf(1 - alpha/2)
p_bar = (p_c + p_v) / 2  # pooled baseline

mde = 0.02               # minimum detectable effect (absolute)

# Required sample size per variant

n_required = (z_alpha**2 * 2 * p_bar * (1 - p_bar)) / (mde**2)
print(f"Required sample per variant: {math.ceil(n_required)}")

```

This ensures experiments meet the 80% power threshold before drawing conclusions.

## Key Source Files

The statistical methodology is defined in two primary locations:

- **[`pm-data-analytics/skills/ab-test-analysis/SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/pm-data-analytics/skills/ab-test-analysis/SKILL.md)**: Declares the complete workflow including Z-tests, confidence intervals, and power analysis formulas (lines 26-56)
- **[`pm-data-analytics/commands/analyze-test.md`](https://github.com/phuryn/pm-skills/blob/main/pm-data-analytics/commands/analyze-test.md)**: Implements the CLI command that triggers the skill and structures the output report

These files establish the contract between the statistical engine and the user-facing interface, ensuring consistent application of hypothesis testing principles across all experiments.

## Summary

- The A/B test analysis skill applies a **two-tailed Z-test** (or chi-squared equivalent) to calculate p-values and 95% confidence intervals for conversion rate differences.
- **Power analysis** using the formula $n = (Z^2 \times 2p(1-p)) / MDE^2$ validates experimental design before analysis begins.
- **Relative lift** calculations quantify practical significance, mapping statistical results to business decisions (Ship, Extend, Stop, Investigate).
- The workflow is implemented in [`pm-data-analytics/skills/ab-test-analysis/SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/pm-data-analytics/skills/ab-test-analysis/SKILL.md) and triggered via the `/analyze-test` command.

## Frequently Asked Questions

### What statistical test does the A/B test analysis skill use for significance testing?

The skill uses a **two-tailed Z-test for proportions** to calculate p-values, which is mathematically equivalent to a chi-squared test for count data. This tests the null hypothesis that conversion rates are equal between control and variant groups, producing a p-value and 95% confidence interval for the difference.

### How does the skill determine if an experiment has enough data?

The skill calculates **statistical power** using the formula $n = (Z_{\alpha/2}^2 \times 2 \times p \times (1-p)) / MDE^2$ to determine the required sample size per variant. It flags experiments as under-powered if they fall below 80% power or lack sufficient duration to detect the minimum detectable effect, preventing premature conclusions.

### What is the difference between statistical and practical significance in this skill?

**Statistical significance** (p-value < 0.05) indicates the observed difference is unlikely due to chance, while **practical significance** measures business impact through relative lift percentage. The skill requires both criteria—significant p-values and meaningful lift—to recommend shipping a variant, ensuring changes are both statistically valid and commercially valuable.

### Can the skill analyze segmented or subgroup data?

Yes, the workflow supports optional segmentation by user attributes or secondary metrics. While the core methodology applies the same Z-test logic to subgroups, the skill emphasizes that segmentation requires sufficient sample sizes within each segment to maintain statistical power and avoid false positives from multiple comparisons.