Statistical Methods for A/B Testing in the ab-test-analysis Skill

The ab-test-analysis skill employs a two-tailed Z-test or chi-squared test for proportions, constructs 95% confidence intervals, and validates statistical power using a binomial sample-size formula to determine experiment significance.

The ab-test-analysis skill in the phuryn/pm-skills repository provides a rigorous statistical framework for evaluating randomized controlled experiments. Defined in pm-data-analytics/skills/ab-test-analysis/SKILL.md, the skill evaluates both statistical significance through hypothesis testing and practical significance through effect size metrics before rendering product recommendations.

Overview of the Statistical Workflow

The skill executes a six-step analytical pipeline that combines frequentist inference with power analysis:

1. Conversion-rate calculation – Computes the proportion of successes for control and variant groups (lines 34‑35).
2. Relative lift – Calculates the percent change of the variant relative to the control (line 35).
3. Statistical test – Applies either a two-tailed Z-test or a chi-squared test to obtain a p‑value (line 36).
4. Confidence interval – Builds a 95 % confidence interval for the difference between the two rates (line 37).
5. Significance decision – Interprets results as significant when p < 0.05 (line 38).
6. Sample-size validation – Flags under-powered tests using the classic binomial power formula (lines 26‑28).

This workflow ensures that decisions are based on both the probability of observed differences occurring by chance and the magnitude of the observed effect.

Core Significance Testing Methods

Two-Tailed Z-Test for Proportions

The primary significance test is a two-tailed Z-test for comparing two proportions, implemented using statsmodels.stats.proportion.proportions_ztest. This test evaluates whether the difference between the control conversion rate (p_control) and variant conversion rate (p_variant) exceeds what would be expected under the null hypothesis of no difference.

As specified in SKILL.md (line 36), the test uses the alternative='two-sided' parameter to detect differences in either direction, protecting against directional bias in the experiment design.

Chi-Squared Test Alternative

The skill documentation notes that a chi-squared test may be used as an alternative to the Z-test (line 36). Both tests are asymptotically equivalent for large samples, but the chi-squared approach is distributionally appropriate when analyzing contingency tables of successes and failures across the two groups.

Effect Size and Practical Significance

Statistical significance alone does not justify a product decision; the skill therefore quantifies practical impact through two key metrics.

Relative Lift Calculation

The skill computes relative lift (line 35) to measure the magnitude of improvement:

$$ \text{Lift} = \frac{p_{\text{variant}} - p_{\text{control}}}{p_{\text{control}}} \times 100 $$

This percentage change contextualizes whether a statistically significant result is large enough to warrant implementation costs.

Confidence Interval Construction

For uncertainty quantification, the skill constructs a 95% confidence interval for the difference between the two rates (line 37). Using the normal approximation method, the interval provides a range of plausible values for the true effect size, allowing stakeholders to assess worst-case and best-case scenarios simultaneously.

Statistical Power and Sample Size Validation

Before interpreting results, the skill validates whether the experiment was adequately powered. It applies the classic binomial sample-size formula (lines 26‑28):

$$ n = \frac{Z_{\alpha/2}^{2} \cdot 2 \cdot p \cdot (1-p)}{\text{MDE}^2} $$

Where:

• $Z_{\alpha/2}$ is the critical value for the desired confidence level (1.96 for 95% confidence)
• $p$ is the pooled probability of conversion
• $\text{MDE}$ is the minimum detectable effect size

Tests with less than 80% power are flagged as under-powered, indicating that the sample size may be insufficient to detect a meaningful difference even if one exists.

Implementation Example

The following Python implementation mirrors the statistical workflow described in SKILL.md:

import pandas as pd
import statsmodels.stats.proportion as smp

# Load raw experiment data (must contain `group` and `converted` columns)

df = pd.read_csv('experiment.csv')

# Aggregate counts

control = df[df.group == 'control']['converted'].sum()
control_n = df[df.group == 'control'].shape[0]
variant = df[df.group == 'variant']['converted'].sum()
variant_n = df[df.group == 'variant'].shape[0]

# Conversion rates

p_control = control / control_n
p_variant = variant / variant_n
lift = (p_variant - p_control) / p_control * 100

# Two-tailed Z-test (proportion comparison)

z_stat, p_val = smp.proportions_ztest([variant, control],
                                      [variant_n, control_n],
                                      alternative='two-sided')

# 95% confidence interval for the difference

ci_low, ci_upp = smp.proportion_confint(variant, variant_n, alpha=0.05, method='normal')
ci_diff = (ci_low - p_control, ci_upp - p_control)

print(f'Control rate: {p_control:.2%}')
print(f'Variant rate: {p_variant:.2%}')
print(f'Relative lift: {lift:.2f}%')
print(f'p-value (Z-test): {p_val:.4f}')
print(f'95% CI for difference: {ci_diff[0]:.2%} – {ci_diff[1]:.2%}')

If p_val < 0.05 and the lift is business-relevant, the skill renders a "Ship it" recommendation; otherwise, it suggests extending, stopping, or investigating the test (see the decision matrix in SKILL.md, lines 49‑55).

Summary

• The ab-test-analysis skill uses two-tailed Z-tests or chi-squared tests to calculate p-values for comparing conversion rates between groups.
• It enforces a 0.05 significance threshold (α = 0.05) and constructs 95% confidence intervals to bound the true effect size.
• Relative lift calculations translate statistical results into business impact percentages.
• A binomial sample-size formula validates that experiments meet the 80% power standard before drawing conclusions.
• All statistical logic is documented in pm-data-analytics/skills/ab-test-analysis/SKILL.md.

Frequently Asked Questions

What significance threshold does the ab-test-analysis skill use?

The skill uses a standard alpha level of 0.05, interpreting results as statistically significant when p < 0.05 (line 38 in SKILL.md). This corresponds to a 95% confidence level, meaning there is less than a 5% probability that the observed difference occurred by random chance alone.

How does the skill calculate required sample size?

The skill applies the classic binomial sample-size formula shown in lines 26‑28 of SKILL.md: $n = (Z_{\alpha/2}^2 \cdot 2 \cdot p \cdot (1-p)) / \text{MDE}^2$. This calculation ensures the experiment can detect the minimum detectable effect (MDE) with at least 80% statistical power, flagging under-powered tests that risk Type II errors.

Can the skill use chi-squared instead of Z-test?

Yes. While the two-tailed Z-test is the primary method for comparing two proportions, the skill documentation explicitly mentions the chi-squared test as an alternative (line 36). Both methods are asymptotically equivalent for large samples, but the chi-squared approach is particularly appropriate when analyzing frequency tables of categorical outcomes.

What practical significance metrics does the skill provide?

Beyond p-values, the skill calculates relative lift to quantify the percentage improvement of the variant over the control (line 35). It also constructs 95% confidence intervals for the rate difference (line 37), allowing stakeholders to evaluate whether the effect size justifies implementation costs even when statistical significance is achieved.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}