How to Perform A/B Test Analysis Using the ab-test-analysis Skill in pm-skills

Use the /analyze-test command to invoke the ab-test-analysis skill, which validates experiment design, calculates statistical significance via generated Python scripts, and returns a markdown report with actionable recommendations like Ship, Extend, Stop, or Investigate.

The pm-skills repository provides a declarative framework for product management automation, offering rigorous statistical capabilities through pure markdown skill definitions. The ab-test-analysis skill, housed in the pm-data-analytics package, enables comprehensive A/B test analysis without embedding executable code directly in the repository. This guide explains how to perform A/B test analysis using the ab-test-analysis skill through interactive commands and programmatic workflows.

How the ab-test-analysis Skill Works

The skill is defined in pm-data-analytics/skills/ab-test-analysis/SKILL.md as a declarative specification that guides the execution engine through a six-step analytical workflow:

1. Collect experiment context – Captures hypothesis, variant names, primary metric, duration, and traffic split.
2. Validate test design – Performs power analysis, checks duration adequacy, detects sample ratio mismatch (SRM), and assesses novelty effects.
3. Compute statistics – Calculates conversion rates, relative lift, two-tailed p-values, and 95% confidence intervals.
4. Check guard-rail metrics – Monitors for adverse effects on revenue, engagement, or performance indicators.
5. Map outcomes to decisions – Categorizes results into Ship, Extend, Stop, or Investigate based on the decision table defined in the skill.
6. Render a markdown report – Generates a ready-to-copy summary containing tables, recommendations, and next-step suggestions.

Because the skill contains no executable logic, the framework delegates statistical computations to a runtime environment that executes generated Python snippets.

Invoking the Skill via the /analyze-test Command

The pm-data-analytics/commands/analyze-test.md file defines the /analyze-test command wrapper that parses user input and forwards it to the ab-test-analysis skill. When invoked, the command:

• Parses raw CSV files, screenshots, or textual summaries provided by the user.
• Passes structured data to the skill validation layer.
• Triggers generation of Python analysis scripts using scipy.stats for z-tests or chi-square calculations.
• Returns a formatted markdown report for documentation or decision logs.

Analyzing Summary Statistics

For quick analysis when you already have aggregate data, pass conversion rates and sample sizes directly:

/analyze-test Control: 4.2% conversion (n=5000), Variant: 4.8% conversion (n=5100)

The engine parses these values, computes the two-tailed p-value and confidence interval, and returns a report such as:

## A/B Test Analysis: My Experiment

**Hypothesis**: New button increases sign-ups
**Duration**: 14 days | **Sample**: 5,000 control / 5,100 variant

| Metric | Control | Variant | Lift | p-value | Significant? |
|--------|---------|---------|------|---------|--------------|
| Sign-up rate | 4.2% | 4.8% | +14.3% | 0.003 | Yes |

**Recommendation**: Ship it — roll out to 100%
**Reasoning**: Statistically and practically significant lift with no guard-rail degradation.

Processing Raw CSV Data

Upload a CSV containing columns user_id, variant, converted, and timestamp, then invoke:

/analyze-test [upload CSV]

The framework loads the CSV, generates a Python script to perform the statistical calculations, and enriches the report with a Sample Size Check section comparing required versus actual sample sizes.

Statistical Implementation and Python Integration

While the repository stores only markdown definitions, the ab-test-analysis skill generates Python code that leverages scipy.stats for rigorous statistical testing. To embed this logic in custom pipelines or dashboards, use the underlying calculation approach:

import pandas as pd
from scipy import stats

df = pd.read_csv('ab_test_results.csv')
control = df[df.variant == 'control']
variant = df[df.variant == 'variant']

p_control = control.converted.mean()
p_variant = variant.converted.mean()
n_control = len(control)
n_variant = len(variant)

# Two-tailed z-test for proportions

pooled = (p_control * n_control + p_variant * n_variant) / (n_control + n_variant)
se = (pooled * (1 - pooled) * (1 / n_control + 1 / n_variant)) ** 0.5
z = (p_variant - p_control) / se
p_val = 2 * (1 - stats.norm.cdf(abs(z)))

print(f'p-value: {p_val:.4f}')

This produces identical statistical values to the skill's automated reports, enabling integration with CI pipelines or external analytics platforms.

Decision Framework: From Data to Action

According to the decision matrix defined in pm-data-analytics/skills/ab-test-analysis/SKILL.md, the skill maps statistical results to four concrete product decisions:

• Ship – Roll out to 100% traffic when results show statistical and practical significance with healthy guard-rail metrics.
• Extend – Continue the test when trending toward significance but sample size is insufficient.
• Stop – Terminate the experiment when showing neutral or negative results with conclusive significance.
• Investigate – Pause for manual review when guard-rail metrics degrade or sample ratio mismatch (SRM) is detected.

Guard-rail metrics ensure that primary metric improvements do not come at the cost of revenue drops, engagement declines, or performance degradation.

Summary

• The /analyze-test command defined in pm-data-analytics/commands/analyze-test.md serves as the primary interface for A/B test analysis.
• The skill specification in pm-data-analytics/skills/ab-test-analysis/SKILL.md validates experimental design and orchestrates statistical calculations.
• Analysis supports both summary statistics and raw CSV inputs, automatically generating Python scripts using scipy.stats for z-tests and confidence intervals.
• Results categorize into four decision outcomes—Ship, Extend, Stop, or Investigate—based on statistical significance, practical lift, and guard-rail metric health.

Frequently Asked Questions

What statistical tests does the ab-test-analysis skill use?

The skill generates Python scripts that perform two-tailed z-tests for comparing conversion rates between variants, utilizing scipy.stats.norm for p-value calculations. For categorical data analysis or contingency tables, it can generate chi-square tests to detect significant distribution differences.

Can I analyze my own CSV data with the ab-test-analysis skill?

Yes, the /analyze-test command accepts CSV uploads containing columns such as user_id, variant, converted, and timestamp. The engine parses this data, validates the experimental design, and produces statistical reports identical to those generated from summary statistics.

How does the skill determine whether to Ship or Stop an experiment?

The decision matrix evaluates statistical significance (typically p < 0.05), practical significance (minimum detectable effect thresholds), and guard-rail health. Ship requires positive lift on primary metrics with no degradation in guard-rail metrics, while Stop triggers on statistically significant negative results or harmful guard-rail effects.

Is the statistical computation performed inside the pm-skills repository?

No, the repository contains only declarative markdown definitions in files like SKILL.md and analyze-test.md. The framework generates Python code at runtime to perform calculations using libraries such as scipy.stats, keeping the repository lightweight while maintaining access to rigorous statistical methods.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}