# Statistical Methods for A/B Testing in the ab-test-analysis Skill

> Discover how the ab-test-analysis skill uses Z-tests, chi-squared tests, and confidence intervals to validate experiment significance and statistical power.

- Repository: [Pawel Huryn/pm-skills](https://github.com/phuryn/pm-skills)
- Tags: deep-dive
- Published: 2026-06-22

---

**The ab-test-analysis skill employs a two-tailed Z-test or chi-squared test for proportions, constructs 95% confidence intervals, and validates statistical power using a binomial sample-size formula to determine experiment significance.**

The `ab-test-analysis` skill in the `phuryn/pm-skills` repository provides a rigorous statistical framework for evaluating randomized controlled experiments. Defined in [`pm-data-analytics/skills/ab-test-analysis/SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/pm-data-analytics/skills/ab-test-analysis/SKILL.md), the skill evaluates both **statistical significance** through hypothesis testing and **practical significance** through effect size metrics before rendering product recommendations.

## Overview of the Statistical Workflow

The skill executes a six-step analytical pipeline that combines frequentist inference with power analysis:

1. **Conversion-rate calculation** – Computes the proportion of successes for control and variant groups (lines 34‑35).
2. **Relative lift** – Calculates the percent change of the variant relative to the control (line 35).
3. **Statistical test** – Applies either a **two-tailed Z-test** or a **chi-squared test** to obtain a p‑value (line 36).
4. **Confidence interval** – Builds a 95 % confidence interval for the difference between the two rates (line 37).
5. **Significance decision** – Interprets results as significant when **p < 0.05** (line 38).
6. **Sample-size validation** – Flags under-powered tests using the classic binomial power formula (lines 26‑28).

This workflow ensures that decisions are based on both the probability of observed differences occurring by chance and the magnitude of the observed effect.

## Core Significance Testing Methods

### Two-Tailed Z-Test for Proportions

The primary significance test is a **two-tailed Z-test for comparing two proportions**, implemented using `statsmodels.stats.proportion.proportions_ztest`. This test evaluates whether the difference between the control conversion rate (`p_control`) and variant conversion rate (`p_variant`) exceeds what would be expected under the null hypothesis of no difference.

As specified in [`SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/SKILL.md) (line 36), the test uses the `alternative='two-sided'` parameter to detect differences in either direction, protecting against directional bias in the experiment design.

### Chi-Squared Test Alternative

The skill documentation notes that a **chi-squared test** may be used as an alternative to the Z-test (line 36). Both tests are asymptotically equivalent for large samples, but the chi-squared approach is distributionally appropriate when analyzing contingency tables of successes and failures across the two groups.

## Effect Size and Practical Significance

Statistical significance alone does not justify a product decision; the skill therefore quantifies practical impact through two key metrics.

### Relative Lift Calculation

The skill computes **relative lift** (line 35) to measure the magnitude of improvement:

$$
\text{Lift} = \frac{p_{\text{variant}} - p_{\text{control}}}{p_{\text{control}}} \times 100
$$

This percentage change contextualizes whether a statistically significant result is large enough to warrant implementation costs.

### Confidence Interval Construction

For uncertainty quantification, the skill constructs a **95% confidence interval** for the difference between the two rates (line 37). Using the normal approximation method, the interval provides a range of plausible values for the true effect size, allowing stakeholders to assess worst-case and best-case scenarios simultaneously.

## Statistical Power and Sample Size Validation

Before interpreting results, the skill validates whether the experiment was adequately powered. It applies the classic binomial sample-size formula (lines 26‑28):

$$
n = \frac{Z_{\alpha/2}^{2} \cdot 2 \cdot p \cdot (1-p)}{\text{MDE}^2}
$$

Where:
- $Z_{\alpha/2}$ is the critical value for the desired confidence level (1.96 for 95% confidence)
- $p$ is the pooled probability of conversion
- $\text{MDE}$ is the minimum detectable effect size

Tests with less than **80% power** are flagged as under-powered, indicating that the sample size may be insufficient to detect a meaningful difference even if one exists.

## Implementation Example

The following Python implementation mirrors the statistical workflow described in [`SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/SKILL.md):

```python
import pandas as pd
import statsmodels.stats.proportion as smp

# Load raw experiment data (must contain `group` and `converted` columns)

df = pd.read_csv('experiment.csv')

# Aggregate counts

control = df[df.group == 'control']['converted'].sum()
control_n = df[df.group == 'control'].shape[0]
variant = df[df.group == 'variant']['converted'].sum()
variant_n = df[df.group == 'variant'].shape[0]

# Conversion rates

p_control = control / control_n
p_variant = variant / variant_n
lift = (p_variant - p_control) / p_control * 100

# Two-tailed Z-test (proportion comparison)

z_stat, p_val = smp.proportions_ztest([variant, control],
                                      [variant_n, control_n],
                                      alternative='two-sided')

# 95% confidence interval for the difference

ci_low, ci_upp = smp.proportion_confint(variant, variant_n, alpha=0.05, method='normal')
ci_diff = (ci_low - p_control, ci_upp - p_control)

print(f'Control rate: {p_control:.2%}')
print(f'Variant rate: {p_variant:.2%}')
print(f'Relative lift: {lift:.2f}%')
print(f'p-value (Z-test): {p_val:.4f}')
print(f'95% CI for difference: {ci_diff[0]:.2%} – {ci_diff[1]:.2%}')

```

If `p_val < 0.05` and the lift is business-relevant, the skill renders a "Ship it" recommendation; otherwise, it suggests extending, stopping, or investigating the test (see the decision matrix in [`SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/SKILL.md), lines 49‑55).

## Summary

- The **ab-test-analysis** skill uses **two-tailed Z-tests** or **chi-squared tests** to calculate p-values for comparing conversion rates between groups.
- It enforces a **0.05 significance threshold** (α = 0.05) and constructs **95% confidence intervals** to bound the true effect size.
- **Relative lift** calculations translate statistical results into business impact percentages.
- A **binomial sample-size formula** validates that experiments meet the 80% power standard before drawing conclusions.
- All statistical logic is documented in [`pm-data-analytics/skills/ab-test-analysis/SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/pm-data-analytics/skills/ab-test-analysis/SKILL.md).

## Frequently Asked Questions

### What significance threshold does the ab-test-analysis skill use?

The skill uses a standard **alpha level of 0.05**, interpreting results as statistically significant when **p < 0.05** (line 38 in [`SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/SKILL.md)). This corresponds to a 95% confidence level, meaning there is less than a 5% probability that the observed difference occurred by random chance alone.

### How does the skill calculate required sample size?

The skill applies the classic binomial sample-size formula shown in lines 26‑28 of [`SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/SKILL.md): $n = (Z_{\alpha/2}^2 \cdot 2 \cdot p \cdot (1-p)) / \text{MDE}^2$. This calculation ensures the experiment can detect the minimum detectable effect (MDE) with at least 80% statistical power, flagging under-powered tests that risk Type II errors.

### Can the skill use chi-squared instead of Z-test?

Yes. While the **two-tailed Z-test** is the primary method for comparing two proportions, the skill documentation explicitly mentions the **chi-squared test** as an alternative (line 36). Both methods are asymptotically equivalent for large samples, but the chi-squared approach is particularly appropriate when analyzing frequency tables of categorical outcomes.

### What practical significance metrics does the skill provide?

Beyond p-values, the skill calculates **relative lift** to quantify the percentage improvement of the variant over the control (line 35). It also constructs **95% confidence intervals** for the rate difference (line 37), allowing stakeholders to evaluate whether the effect size justifies implementation costs even when statistical significance is achieved.