# How to Analyze A/B Test Results with Statistical Significance Using PM Skills

> Master A/B test results analysis with statistical significance. Use PM Skills to validate experiments, calculate p-values, and make data-driven product decisions. Ship, Extend, Stop, or Investigate with confidence.

- Repository: [Pawel Huryn/pm-skills](https://github.com/phuryn/pm-skills)
- Tags: how-to-guide
- Published: 2026-06-17

---

**PM Skills provides a dedicated A/B-test analysis skill that validates experiment design, calculates two-tailed z-test p-values with 95% confidence intervals, and returns product decisions (Ship/Extend/Stop/Investigate) through the `/analyze-test` command.**

The `phuryn/pm-skills` repository ships with a complete analytics toolkit for product managers who need to evaluate experiments without writing statistical boilerplate. The A/B test analysis capability lives in [`pm-data-analytics/skills/ab-test-analysis/SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/pm-data-analytics/skills/ab-test-analysis/SKILL.md) and is exposed via the `/analyze-test` command defined in [`pm-data-analytics/commands/analyze-test.md`](https://github.com/phuryn/pm-skills/blob/main/pm-data-analytics/commands/analyze-test.md). This skill guides you through a rigorous four-phase workflow that ensures your conclusions are statistically sound and actionable.

## The PM Skills A/B Test Analysis Workflow

The skill breaks down experiment evaluation into four logical phases, each implemented as plain-text instructions that the PM Skills engine parses and executes.

### Phase 1: Capture Experiment Context

First, the skill collects essential metadata: your hypothesis, variant names, primary and guardrail metrics, test duration, and traffic split. This context anchors the statistical analysis to specific product outcomes rather than abstract numbers.

### Phase 2: Validate Test Design

Before calculating significance, the skill automatically checks **sample-size adequacy** using standard power-analysis formulas, validates **test duration** (enforcing minimum 1-2 business cycles), detects **sample-ratio-mismatch** to verify randomization, and flags potential **novelty effects**. These validation rules are encoded directly in [`pm-data-analytics/skills/ab-test-analysis/SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/pm-data-analytics/skills/ab-test-analysis/SKILL.md).

### Phase 3: Calculate Statistical Significance

This phase computes conversion rates, relative lift, two-tailed z-test (or chi-squared) p-values, and 95% confidence intervals. When you supply raw data, the skill generates a Python script that imports `pandas` and `scipy.stats` to perform these calculations locally.

### Phase 4: Interpret Results and Recommend Actions

Based on the significance matrix, the skill returns a clear product decision: **Ship**, **Extend**, **Stop**, or **Investigate**. Each recommendation includes business-impact estimates and next-step suggestions. The decision table is built into the skill file and maps statistical outcomes to product actions.

## Invoking the A/B Test Analysis Command

The `/analyze-test` command accepts multiple input formats and forwards them to the skill engine.

### Simple Invocation with Summary Statistics

Pass conversion rates and sample sizes directly:

```text
/analyze-test Control: 4.2% conversion (n=5000), Variant: 4.8% conversion (n=5100)

```

The command parses these numbers, runs the built-in statistical routine, and returns a markdown summary following the template in [`analyze-test.md`](https://github.com/phuryn/pm-skills/blob/main/analyze-test.md).

### Supplying Raw CSV Data

Attach a CSV file for deeper analysis:

```text
/analyze-test

# attach file: ab_test_results.csv

```

The CSV must contain columns `user_id`, `variant`, and `converted`. The skill then generates and executes a Python script to compute exact statistics.

## Understanding the Statistical Calculations

When raw data is provided, PM Skills emits a Python script that performs the following calculations using `pandas` and `scipy.stats`:

```python
import pandas as pd
from scipy import stats

df = pd.read_csv('ab_test_results.csv')
control = df[df.variant == 'control']
variant = df[df.variant == 'variant']

# conversion rates

p_control = control.converted.mean()
p_variant = variant.converted.mean()

# lift calculation

lift = (p_variant - p_control) / p_control * 100

# two-tailed z-test

n_control = len(control)
n_variant = len(variant)
p_pool = (control.converted.sum() + variant.converted.sum()) / (n_control + n_variant)
z = (p_variant - p_control) / ((p_pool * (1 - p_pool) * (1/n_control + 1/n_variant)) ** 0.5)
p_value = 2 * (1 - stats.norm.cdf(abs(z)))

# 95% confidence interval for the difference

se = (p_pool * (1 - p_pool) * (1/n_control + 1/n_variant)) ** 0.5
ci_low = (p_variant - p_control) - 1.96 * se
ci_high = (p_variant - p_control) + 1.96 * se

print(f"Control CR: {p_control:.3%} (n={n_control})")
print(f"Variant CR: {p_variant:.3%} (n={n_variant})")
print(f"Lift: {lift:.2f}%")
print(f"P-value: {p_value:.4f}")
print(f"95% CI: [{ci_low:.3%}, {ci_high:.3%}]")

```

Running this script yields the precise metrics that populate the skill's final recommendation table.

## Interpreting the Output

The skill returns a structured markdown report that you can paste directly into product docs or Slack:

```markdown

## A/B Test Results: Checkout CTA Experiment

**Hypothesis**: New CTA increases checkout conversion.

| Metric   | Control | Variant | Lift  | p-value | Significant? |
|----------|---------|---------|-------|---------|--------------|
| Conversion | 4.20% | 4.80% | +14.3% | 0.0012 | Yes |
| Revenue (guardrail) | $1.20 | $1.18 | -1.7% | – | No concern |

**Recommendation:** **SHIP** – lift is statistically and practically significant, guardrails unchanged.

```

This output follows the template from [`pm-data-analytics/skills/ab-test-analysis/SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/pm-data-analytics/skills/ab-test-analysis/SKILL.md) and includes the decision matrix that maps statistical results to product actions.

## Summary

- **Location**: The A/B test analysis skill resides in [`pm-data-analytics/skills/ab-test-analysis/SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/pm-data-analytics/skills/ab-test-analysis/SKILL.md) and is invoked via `/analyze-test` defined in [`pm-data-analytics/commands/analyze-test.md`](https://github.com/phuryn/pm-skills/blob/main/pm-data-analytics/commands/analyze-test.md).
- **Validation**: The skill automatically checks sample size, duration, randomization, and novelty effects before calculating significance.
- **Statistics**: It calculates two-tailed z-test p-values, 95% confidence intervals, and relative lift using standard formulas implemented in generated Python scripts.
- **Decisions**: Output maps to four actions—Ship, Extend, Stop, or Investigate—based on statistical significance and guardrail metrics.
- **Integration**: Because the skill is pure markdown with embedded Python generation, it works in any chat-oriented product (Slack bots, CLI wrappers) requiring only a Python interpreter for calculations.

## Frequently Asked Questions

### How does PM Skills validate that my A/B test ran long enough?

The skill checks test duration against the minimum threshold of 1-2 full business cycles to account for day-of-week effects and user behavior variations. This validation occurs in Phase 2 of the workflow defined in [`pm-data-analytics/skills/ab-test-analysis/SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/pm-data-analytics/skills/ab-test-analysis/SKILL.md). If your test duration is insufficient, the skill flags this before calculating significance to prevent false conclusions.

### What statistical test does PM Skills use for calculating significance?

The skill uses a **two-tailed z-test** for comparing proportions (conversion rates) or chi-squared tests when appropriate, depending on your data format. These calculations are implemented in the Python scripts generated on-demand when you supply raw CSV data. The formulas account for pooled variance and include 95% confidence intervals using the standard 1.96 critical value.

### Can I use PM Skills A/B test analysis without installing Python?

Yes, if you provide summary statistics (conversion rates and sample sizes) directly to the `/analyze-test` command, the skill performs calculations internally without requiring Python. However, if you upload raw CSV data, the skill generates a Python script that requires `pandas` and `scipy` to compute exact p-values and confidence intervals. The only runtime requirement is a Python interpreter for generated scripts.

### How does the skill decide between Ship, Extend, Stop, or Investigate?

The decision matrix in [`pm-data-analytics/skills/ab-test-analysis/SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/pm-data-analytics/skills/ab-test-analysis/SKILL.md) evaluates statistical significance (p-value < 0.05), practical significance (minimum detectable effect), and guardrail metrics. **Ship** means the primary metric improved significantly with no guardrail violations. **Extend** suggests running longer if significance is borderline. **Stop** terminates underperforming variants. **Investigate** triggers when data quality issues or conflicting metrics appear.