How to Analyze A/B Test Results with Statistical Significance Using PM Skills

PM Skills provides a dedicated A/B-test analysis skill that validates experiment design, calculates two-tailed z-test p-values with 95% confidence intervals, and returns product decisions (Ship/Extend/Stop/Investigate) through the /analyze-test command.

The phuryn/pm-skills repository ships with a complete analytics toolkit for product managers who need to evaluate experiments without writing statistical boilerplate. The A/B test analysis capability lives in pm-data-analytics/skills/ab-test-analysis/SKILL.md and is exposed via the /analyze-test command defined in pm-data-analytics/commands/analyze-test.md. This skill guides you through a rigorous four-phase workflow that ensures your conclusions are statistically sound and actionable.

The PM Skills A/B Test Analysis Workflow

The skill breaks down experiment evaluation into four logical phases, each implemented as plain-text instructions that the PM Skills engine parses and executes.

Phase 1: Capture Experiment Context

First, the skill collects essential metadata: your hypothesis, variant names, primary and guardrail metrics, test duration, and traffic split. This context anchors the statistical analysis to specific product outcomes rather than abstract numbers.

Phase 2: Validate Test Design

Before calculating significance, the skill automatically checks sample-size adequacy using standard power-analysis formulas, validates test duration (enforcing minimum 1-2 business cycles), detects sample-ratio-mismatch to verify randomization, and flags potential novelty effects. These validation rules are encoded directly in pm-data-analytics/skills/ab-test-analysis/SKILL.md.

Phase 3: Calculate Statistical Significance

This phase computes conversion rates, relative lift, two-tailed z-test (or chi-squared) p-values, and 95% confidence intervals. When you supply raw data, the skill generates a Python script that imports pandas and scipy.stats to perform these calculations locally.

Phase 4: Interpret Results and Recommend Actions

Based on the significance matrix, the skill returns a clear product decision: Ship, Extend, Stop, or Investigate. Each recommendation includes business-impact estimates and next-step suggestions. The decision table is built into the skill file and maps statistical outcomes to product actions.

Invoking the A/B Test Analysis Command

The /analyze-test command accepts multiple input formats and forwards them to the skill engine.

Simple Invocation with Summary Statistics

Pass conversion rates and sample sizes directly:

/analyze-test Control: 4.2% conversion (n=5000), Variant: 4.8% conversion (n=5100)

The command parses these numbers, runs the built-in statistical routine, and returns a markdown summary following the template in analyze-test.md.

Supplying Raw CSV Data

Attach a CSV file for deeper analysis:

/analyze-test

# attach file: ab_test_results.csv

The CSV must contain columns user_id, variant, and converted. The skill then generates and executes a Python script to compute exact statistics.

Understanding the Statistical Calculations

When raw data is provided, PM Skills emits a Python script that performs the following calculations using pandas and scipy.stats:

import pandas as pd
from scipy import stats

df = pd.read_csv('ab_test_results.csv')
control = df[df.variant == 'control']
variant = df[df.variant == 'variant']

# conversion rates

p_control = control.converted.mean()
p_variant = variant.converted.mean()

# lift calculation

lift = (p_variant - p_control) / p_control * 100

# two-tailed z-test

n_control = len(control)
n_variant = len(variant)
p_pool = (control.converted.sum() + variant.converted.sum()) / (n_control + n_variant)
z = (p_variant - p_control) / ((p_pool * (1 - p_pool) * (1/n_control + 1/n_variant)) ** 0.5)
p_value = 2 * (1 - stats.norm.cdf(abs(z)))

# 95% confidence interval for the difference

se = (p_pool * (1 - p_pool) * (1/n_control + 1/n_variant)) ** 0.5
ci_low = (p_variant - p_control) - 1.96 * se
ci_high = (p_variant - p_control) + 1.96 * se

print(f"Control CR: {p_control:.3%} (n={n_control})")
print(f"Variant CR: {p_variant:.3%} (n={n_variant})")
print(f"Lift: {lift:.2f}%")
print(f"P-value: {p_value:.4f}")
print(f"95% CI: [{ci_low:.3%}, {ci_high:.3%}]")

Running this script yields the precise metrics that populate the skill's final recommendation table.

Interpreting the Output

The skill returns a structured markdown report that you can paste directly into product docs or Slack:

## A/B Test Results: Checkout CTA Experiment

**Hypothesis**: New CTA increases checkout conversion.

| Metric   | Control | Variant | Lift  | p-value | Significant? |
|----------|---------|---------|-------|---------|--------------|
| Conversion | 4.20% | 4.80% | +14.3% | 0.0012 | Yes |
| Revenue (guardrail) | $1.20 | $1.18 | -1.7% | – | No concern |

**Recommendation:** **SHIP** – lift is statistically and practically significant, guardrails unchanged.

This output follows the template from pm-data-analytics/skills/ab-test-analysis/SKILL.md and includes the decision matrix that maps statistical results to product actions.

Summary

• Location: The A/B test analysis skill resides in pm-data-analytics/skills/ab-test-analysis/SKILL.md and is invoked via /analyze-test defined in pm-data-analytics/commands/analyze-test.md.
• Validation: The skill automatically checks sample size, duration, randomization, and novelty effects before calculating significance.
• Statistics: It calculates two-tailed z-test p-values, 95% confidence intervals, and relative lift using standard formulas implemented in generated Python scripts.
• Decisions: Output maps to four actions—Ship, Extend, Stop, or Investigate—based on statistical significance and guardrail metrics.
• Integration: Because the skill is pure markdown with embedded Python generation, it works in any chat-oriented product (Slack bots, CLI wrappers) requiring only a Python interpreter for calculations.

Frequently Asked Questions

How does PM Skills validate that my A/B test ran long enough?

The skill checks test duration against the minimum threshold of 1-2 full business cycles to account for day-of-week effects and user behavior variations. This validation occurs in Phase 2 of the workflow defined in pm-data-analytics/skills/ab-test-analysis/SKILL.md. If your test duration is insufficient, the skill flags this before calculating significance to prevent false conclusions.

What statistical test does PM Skills use for calculating significance?

The skill uses a two-tailed z-test for comparing proportions (conversion rates) or chi-squared tests when appropriate, depending on your data format. These calculations are implemented in the Python scripts generated on-demand when you supply raw CSV data. The formulas account for pooled variance and include 95% confidence intervals using the standard 1.96 critical value.

Can I use PM Skills A/B test analysis without installing Python?

Yes, if you provide summary statistics (conversion rates and sample sizes) directly to the /analyze-test command, the skill performs calculations internally without requiring Python. However, if you upload raw CSV data, the skill generates a Python script that requires pandas and scipy to compute exact p-values and confidence intervals. The only runtime requirement is a Python interpreter for generated scripts.

How does the skill decide between Ship, Extend, Stop, or Investigate?

The decision matrix in pm-data-analytics/skills/ab-test-analysis/SKILL.md evaluates statistical significance (p-value < 0.05), practical significance (minimum detectable effect), and guardrail metrics. Ship means the primary metric improved significantly with no guardrail violations. Extend suggests running longer if significance is borderline. Stop terminates underperforming variants. Investigate triggers when data quality issues or conflicting metrics appear.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}