A/B Test Analysis Skill Statistical Methods: Z-Tests, Power Analysis, and Effect Size

The A/B test analysis skill in the phuryn/pm-skills repository employs a six-step statistical workflow combining power analysis, two-tailed Z-tests for proportions, confidence intervals, and relative lift calculations to validate experiments and drive product decisions.

The A/B test analysis skill provides a structured framework for evaluating product experiments within the phuryn/pm-skills repository. This open-source implementation codifies standard statistical practices into an executable workflow that transforms raw conversion data into actionable recommendations through rigorous hypothesis testing and effect size estimation.

Statistical Workflow Overview

The skill implements a sequential analytical pipeline defined in pm-data-analytics/skills/ab-test-analysis/SKILL.md. Each step applies specific statistical techniques to ensure experimental validity and practical relevance.

Step 1: Validate Test Design with Power Analysis

Before analyzing results, the skill verifies whether the experiment was properly powered. It calculates the required sample size using the formula:

$n = (Z_{\alpha/2}^2 \times 2 \times p \times (1-p)) / MDE^2$

Where:

• $Z_{\alpha/2}$ represents the critical value for the desired confidence level (typically 1.96 for 95% confidence)
• $p$ denotes the pooled baseline conversion rate
• $MDE$ specifies the minimum detectable effect

According to the source code at lines 26-30, the skill flags under-powered tests that fall below 80% power or lack sufficient duration, preventing premature analysis of inconclusive data.

Step 2: Calculate Conversion Rates

The skill computes conversion rates as simple proportions for both control and variant groups:

$p = \text{conversions} / \text{sample size}$

This calculation establishes the baseline metrics required for subsequent significance testing. The workflow processes raw CSV data containing user-level conversion events, aggregating them by variant assignment before proceeding to comparative analysis.

Step 3: Test Statistical Significance

To determine whether observed differences exceed random variation, the skill implements a two-tailed Z-test for difference in proportions (equivalent to a chi-squared test for count data). As specified in SKILL.md lines 33-38, the analysis yields:

• P-value: Calculated as $2 \times (1 - \text{norm.cdf}(|z|))$ where $z = (p_v - p_c) / SE$
• Standard Error: Computed using the pooled proportion: $\sqrt{p_{pool}(1-p_{pool})(1/n_c + 1/n_v)}$
• 95% Confidence Interval: Absolute difference $\pm 1.96 \times SE$

This approach follows the frequentist hypothesis testing framework, rejecting the null hypothesis when $p < 0.05$.

Step 4: Assess Practical Significance

Statistical significance does not guarantee business value. The skill calculates relative lift to quantify practical impact:

$\text{Lift} = (\text{variant rate} - \text{control rate}) / \text{control rate} \times 100%$

Lines 34-39 of SKILL.md indicate that this metric is compared against guard-rail thresholds to determine whether the observed effect size warrants implementation, regardless of statistical significance.

Step 5: Segment and Guard-Rail Analysis

For deeper insights, the workflow supports optional segmentation of results by user attributes or secondary metrics. While no new statistical tests are mandated, the same Z-test or chi-squared methodology applies to subgroup analyses.

Step 6: Decision Matrix

The final stage maps statistical outputs to concrete product actions. As defined in lines 49-56, the skill categorizes outcomes into four decisions:

• Ship: Statistically significant positive lift with adequate power
• Extend: Under-powered but promising trend requiring additional sample
• Stop: Significant negative impact or neutral results with sufficient power
• Investigate: Anomalous patterns or guard-rail violations requiring manual review

Python Implementation Examples

The /analyze-test command orchestrates these calculations, generating Python code that leverages pandas and scipy.stats. Below is the complete analytical pipeline:

import pandas as pd
import numpy as np
import scipy.stats as st

# Load experiment data

df = pd.read_csv("test_results.csv")

# Aggregate conversion counts by variant

summary = df.groupby("variant")["converted"].agg(["sum", "count"]).reset_index()
control = summary[summary["variant"] == "control"]
variant = summary[summary["variant"] == "variant"]

# Extract counts

c_conv, c_n = control["sum"].iloc[0], control["count"].iloc[0]
v_conv, v_n = variant["sum"].iloc[0], variant["count"].iloc[0]

# Calculate conversion rates

p_c = c_conv / c_n
p_v = v_conv / v_n

# Compute relative lift

lift = (p_v - p_c) / p_c * 100

# Two-tailed Z-test for proportions

pooled = (c_conv + v_conv) / (c_n + v_n)
se = np.sqrt(pooled * (1 - pooled) * (1/c_n + 1/v_n))
z = (p_v - p_c) / se
p_value = 2 * (1 - st.norm.cdf(abs(z)))

# 95% Confidence interval for difference

ci_margin = 1.96 * se
ci_low = (p_v - p_c) - ci_margin
ci_high = (p_v - p_c) + ci_margin

print(f"Control: {p_c:.2%} (n={c_n})")
print(f"Variant: {p_v:.2%} (n={v_n})")
print(f"Relative lift: {lift:.2f}%")
print(f"p-value: {p_value:.4f}")
print(f"95% CI: [{ci_low:.3%}, {ci_high:.3%}]")

Power Analysis Verification

To validate the experimental design prior to analysis, the skill implements the power calculation formula referenced in the documentation:

import math

# Parameters

alpha = 0.05
z_alpha = st.norm.ppf(1 - alpha/2)
p_bar = (p_c + p_v) / 2  # pooled baseline

mde = 0.02               # minimum detectable effect (absolute)

# Required sample size per variant

n_required = (z_alpha**2 * 2 * p_bar * (1 - p_bar)) / (mde**2)
print(f"Required sample per variant: {math.ceil(n_required)}")

This ensures experiments meet the 80% power threshold before drawing conclusions.

Key Source Files

The statistical methodology is defined in two primary locations:

• pm-data-analytics/skills/ab-test-analysis/SKILL.md: Declares the complete workflow including Z-tests, confidence intervals, and power analysis formulas (lines 26-56)
• pm-data-analytics/commands/analyze-test.md: Implements the CLI command that triggers the skill and structures the output report

These files establish the contract between the statistical engine and the user-facing interface, ensuring consistent application of hypothesis testing principles across all experiments.

Summary

• The A/B test analysis skill applies a two-tailed Z-test (or chi-squared equivalent) to calculate p-values and 95% confidence intervals for conversion rate differences.
• Power analysis using the formula $n = (Z^2 \times 2p(1-p)) / MDE^2$ validates experimental design before analysis begins.
• Relative lift calculations quantify practical significance, mapping statistical results to business decisions (Ship, Extend, Stop, Investigate).
• The workflow is implemented in pm-data-analytics/skills/ab-test-analysis/SKILL.md and triggered via the /analyze-test command.

Frequently Asked Questions

What statistical test does the A/B test analysis skill use for significance testing?

The skill uses a two-tailed Z-test for proportions to calculate p-values, which is mathematically equivalent to a chi-squared test for count data. This tests the null hypothesis that conversion rates are equal between control and variant groups, producing a p-value and 95% confidence interval for the difference.

How does the skill determine if an experiment has enough data?

The skill calculates statistical power using the formula $n = (Z_{\alpha/2}^2 \times 2 \times p \times (1-p)) / MDE^2$ to determine the required sample size per variant. It flags experiments as under-powered if they fall below 80% power or lack sufficient duration to detect the minimum detectable effect, preventing premature conclusions.

What is the difference between statistical and practical significance in this skill?

Statistical significance (p-value < 0.05) indicates the observed difference is unlikely due to chance, while practical significance measures business impact through relative lift percentage. The skill requires both criteria—significant p-values and meaningful lift—to recommend shipping a variant, ensuring changes are both statistically valid and commercially valuable.

Can the skill analyze segmented or subgroup data?

Yes, the workflow supports optional segmentation by user attributes or secondary metrics. While the core methodology applies the same Z-test logic to subgroups, the skill emphasizes that segmentation requires sufficient sample sizes within each segment to maintain statistical power and avoid false positives from multiple comparisons.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}