# How to Perform A/B Test Analysis Using the ab-test-analysis Skill in pm-skills

> Learn A/B test analysis using the ab-test-analysis skill in pm-skills. Analyze experiments, get statistical significance, and receive actionable recommendations with markdown reports.

- Repository: [Pawel Huryn/pm-skills](https://github.com/phuryn/pm-skills)
- Tags: how-to-guide
- Published: 2026-06-27

---

**Use the `/analyze-test` command to invoke the ab-test-analysis skill, which validates experiment design, calculates statistical significance via generated Python scripts, and returns a markdown report with actionable recommendations like Ship, Extend, Stop, or Investigate.**

The **pm-skills** repository provides a declarative framework for product management automation, offering rigorous statistical capabilities through pure markdown skill definitions. The **ab-test-analysis** skill, housed in the `pm-data-analytics` package, enables comprehensive A/B test analysis without embedding executable code directly in the repository. This guide explains how to perform A/B test analysis using the ab-test-analysis skill through interactive commands and programmatic workflows.

## How the ab-test-analysis Skill Works

The skill is defined in [`pm-data-analytics/skills/ab-test-analysis/SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/pm-data-analytics/skills/ab-test-analysis/SKILL.md) as a declarative specification that guides the execution engine through a six-step analytical workflow:

1. **Collect experiment context** – Captures hypothesis, variant names, primary metric, duration, and traffic split.
2. **Validate test design** – Performs power analysis, checks duration adequacy, detects sample ratio mismatch (SRM), and assesses novelty effects.
3. **Compute statistics** – Calculates conversion rates, relative lift, two-tailed p-values, and 95% confidence intervals.
4. **Check guard-rail metrics** – Monitors for adverse effects on revenue, engagement, or performance indicators.
5. **Map outcomes to decisions** – Categorizes results into **Ship**, **Extend**, **Stop**, or **Investigate** based on the decision table defined in the skill.
6. **Render a markdown report** – Generates a ready-to-copy summary containing tables, recommendations, and next-step suggestions.

Because the skill contains no executable logic, the framework delegates statistical computations to a runtime environment that executes generated Python snippets.

## Invoking the Skill via the /analyze-test Command

The [`pm-data-analytics/commands/analyze-test.md`](https://github.com/phuryn/pm-skills/blob/main/pm-data-analytics/commands/analyze-test.md) file defines the `/analyze-test` command wrapper that parses user input and forwards it to the ab-test-analysis skill. When invoked, the command:

- Parses raw CSV files, screenshots, or textual summaries provided by the user.
- Passes structured data to the skill validation layer.
- Triggers generation of Python analysis scripts using `scipy.stats` for z-tests or chi-square calculations.
- Returns a formatted markdown report for documentation or decision logs.

### Analyzing Summary Statistics

For quick analysis when you already have aggregate data, pass conversion rates and sample sizes directly:

```text
/analyze-test Control: 4.2% conversion (n=5000), Variant: 4.8% conversion (n=5100)

```

The engine parses these values, computes the two-tailed p-value and confidence interval, and returns a report such as:

```markdown

## A/B Test Analysis: My Experiment

**Hypothesis**: New button increases sign-ups
**Duration**: 14 days | **Sample**: 5,000 control / 5,100 variant

| Metric | Control | Variant | Lift | p-value | Significant? |
|--------|---------|---------|------|---------|--------------|
| Sign-up rate | 4.2% | 4.8% | +14.3% | 0.003 | Yes |

**Recommendation**: Ship it — roll out to 100%
**Reasoning**: Statistically and practically significant lift with no guard-rail degradation.

```

### Processing Raw CSV Data

Upload a CSV containing columns `user_id`, `variant`, `converted`, and `timestamp`, then invoke:

```text
/analyze-test [upload CSV]

```

The framework loads the CSV, generates a Python script to perform the statistical calculations, and enriches the report with a **Sample Size Check** section comparing required versus actual sample sizes.

## Statistical Implementation and Python Integration

While the repository stores only markdown definitions, the ab-test-analysis skill generates Python code that leverages `scipy.stats` for rigorous statistical testing. To embed this logic in custom pipelines or dashboards, use the underlying calculation approach:

```python
import pandas as pd
from scipy import stats

df = pd.read_csv('ab_test_results.csv')
control = df[df.variant == 'control']
variant = df[df.variant == 'variant']

p_control = control.converted.mean()
p_variant = variant.converted.mean()
n_control = len(control)
n_variant = len(variant)

# Two-tailed z-test for proportions

pooled = (p_control * n_control + p_variant * n_variant) / (n_control + n_variant)
se = (pooled * (1 - pooled) * (1 / n_control + 1 / n_variant)) ** 0.5
z = (p_variant - p_control) / se
p_val = 2 * (1 - stats.norm.cdf(abs(z)))

print(f'p-value: {p_val:.4f}')

```

This produces identical statistical values to the skill's automated reports, enabling integration with CI pipelines or external analytics platforms.

## Decision Framework: From Data to Action

According to the decision matrix defined in [`pm-data-analytics/skills/ab-test-analysis/SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/pm-data-analytics/skills/ab-test-analysis/SKILL.md), the skill maps statistical results to four concrete product decisions:

- **Ship** – Roll out to 100% traffic when results show statistical and practical significance with healthy guard-rail metrics.
- **Extend** – Continue the test when trending toward significance but sample size is insufficient.
- **Stop** – Terminate the experiment when showing neutral or negative results with conclusive significance.
- **Investigate** – Pause for manual review when guard-rail metrics degrade or sample ratio mismatch (SRM) is detected.

Guard-rail metrics ensure that primary metric improvements do not come at the cost of revenue drops, engagement declines, or performance degradation.

## Summary

- The `/analyze-test` command defined in [`pm-data-analytics/commands/analyze-test.md`](https://github.com/phuryn/pm-skills/blob/main/pm-data-analytics/commands/analyze-test.md) serves as the primary interface for A/B test analysis.
- The skill specification in [`pm-data-analytics/skills/ab-test-analysis/SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/pm-data-analytics/skills/ab-test-analysis/SKILL.md) validates experimental design and orchestrates statistical calculations.
- Analysis supports both summary statistics and raw CSV inputs, automatically generating Python scripts using `scipy.stats` for z-tests and confidence intervals.
- Results categorize into four decision outcomes—Ship, Extend, Stop, or Investigate—based on statistical significance, practical lift, and guard-rail metric health.

## Frequently Asked Questions

### What statistical tests does the ab-test-analysis skill use?

The skill generates Python scripts that perform **two-tailed z-tests** for comparing conversion rates between variants, utilizing `scipy.stats.norm` for p-value calculations. For categorical data analysis or contingency tables, it can generate chi-square tests to detect significant distribution differences.

### Can I analyze my own CSV data with the ab-test-analysis skill?

Yes, the `/analyze-test` command accepts CSV uploads containing columns such as `user_id`, `variant`, `converted`, and `timestamp`. The engine parses this data, validates the experimental design, and produces statistical reports identical to those generated from summary statistics.

### How does the skill determine whether to Ship or Stop an experiment?

The decision matrix evaluates **statistical significance** (typically p < 0.05), **practical significance** (minimum detectable effect thresholds), and **guard-rail health**. Ship requires positive lift on primary metrics with no degradation in guard-rail metrics, while Stop triggers on statistically significant negative results or harmful guard-rail effects.

### Is the statistical computation performed inside the pm-skills repository?

No, the repository contains only declarative markdown definitions in files like [`SKILL.md`](https://github.com/phuryn/pm-skills/blob/main/SKILL.md) and [`analyze-test.md`](https://github.com/phuryn/pm-skills/blob/main/analyze-test.md). The framework generates Python code at runtime to perform calculations using libraries such as `scipy.stats`, keeping the repository lightweight while maintaining access to rigorous statistical methods.