How to Set Up A/B Test Analysis Using the pm-data-analytics Plugin

To set up A/B test analysis using the pm-data-analytics plugin, install the plugin via Claude Code or Claude Cowork, then invoke the /analyze-test slash-command with your experiment metrics to generate a statistical significance report with concrete ship recommendations.

The pm-data-analytics plugin, available in the phuryn/pm-skills repository, bundles three data-analytics capabilities including dedicated A/B test analysis functionality. This plugin enables product managers to validate experimental design, calculate statistical significance, and receive actionable recommendations without writing statistical code. By leveraging the ab-test-analysis skill through the integrated slash-command, you can convert raw experiment data into validated product decisions.

Installing the pm-data-analytics Plugin

The plugin ships as part of the pm-skills marketplace and supports both Claude Cowork (UI) and Claude Code (CLI) installations.

For Claude Code users, run the two-step installation sequence:

claude plugin marketplace add phuryn/pm-skills
claude plugin install pm-data-analytics@pm-skills

For Claude Cowork, add the marketplace in the UI settings, then enable the pm-data-analytics plugin from the plugin directory.

After installation, verify the plugin metadata in pm-data-analytics/.claude-plugin/plugin.json, which defines the plugin version, author, and bundled skills.

Understanding the A/B Test Skill Architecture

The core logic resides in pm-data-analytics/skills/ab-test-analysis/SKILL.md. This skill definition orchestrates the entire analysis pipeline through four distinct phases:

Experiment Validation

The skill first validates your test setup by checking sample size adequacy, experiment duration, randomisation integrity, and novelty effects (lines 25-32). This prevents interpreting underpowered or biased experiments.

Statistical Calculations

Upon validation, the skill computes conversion rates, relative lift, p-values (using z-test or χ² tests), 95% confidence intervals, and practical significance thresholds (lines 33-38). If you provide raw data rather than summary statistics, the skill automatically generates a Python script to perform these calculations (line 74).

Decision Framework

The skill maps statistical outcomes to product decisions through a decision table (lines 49-55) that translates results into ship, extend, stop, or investigate recommendations based on significance levels, power analysis, and business guardrails.

Report Generation

Finally, the skill outputs a markdown report template (lines 58-72) containing the executive summary, statistical details, business impact estimates, and follow-up actions—ready for stakeholder distribution.

Running the /analyze-test Command

The public interface for A/B test analysis is defined in pm-data-analytics/commands/analyze-test.md. This slash-command wraps the ab-test-analysis skill and standardizes the input/output flow.

Supported Input Formats

The command accepts four data formats (lines 22-27):

• Summary statistics: Pre-calculated conversion rates and sample sizes
• Raw CSV: User-level data with variant assignments
• Screenshot: Uploaded images of experiment dashboards
• Free-form description: Natural language experiment summaries

Analysis Workflow

When you invoke /analyze-test, the command executes a three-stage pipeline (lines 28-40):

1. Design Validation: Performs power analysis, checks duration adequacy, and scans for sample ratio mismatch (SRM) or external confounding factors (lines 28-35)
2. Skill Execution: Calls the ab-test-analysis skill to compute significance metrics and generate recommendations (line 40)
3. Structured Output: Returns a complete markdown report including statistical findings, business impact projections, caveats, and next-step suggestions (lines 48-94)

Typical invocation syntax:

/analyze-test Control: 4.2% conversion (n=5000), Variant: 4.8% conversion (n=5100)

Complete Example: Analyzing a Checkout Flow Experiment

Below is a concrete interaction demonstrating the end-to-end flow:

User: /analyze-test Control: 2.5% conversion (n=8000), Variant: 3.1% conversion (n=7950)

Claude:

## A/B Test Analysis: Checkout Flow Experiment

**Date**: 2026-06-16  
**Test duration**: 21 days  
**Total sample**: 15,950 users  

### Results Summary

| Variant | Sample | Metric      | Rate | 95% CI |
|---------|--------|-------------|------|--------|
| Control | 8,000  | Conversion  | 2.5% | 2.3%-2.7% |
| Variant | 7,950  | Conversion  | 3.1% | 2.9%-3.3% |

### Statistical Analysis

- **Relative lift**: +24% (0.6% ± 0.12%)
- **P-value**: 0.0012  
- **Statistically significant**: Yes (α = 0.05)  
- **Minimum detectable effect**: 0.5% (80% power)

### Sample Size Check

- **Required sample**: 7,500 per variant (for 0.5% MDE)  
- **Actual sample**: 7,950 / 8,000  
- **Verdict**: Sufficiently powered

### Decision

**Recommendation: SHIP**  
Reasoning: The lift is statistically significant and exceeds the minimum detectable effect, with no guard-rail degradation mentioned.

### Business Impact Estimate

- **Expected impact**: +0.6% conversion → ~$45k incremental revenue per month (assuming $7M baseline).

### Caveats

- No segment-level breakdown yet – consider analyzing power users separately.

### Follow-Up

- Set up a post-launch monitoring dashboard (use `/write-query` to generate the SQL).

Extending Your Analysis

Beyond basic significance testing, the pm-data-analytics plugin supports advanced analytical patterns:

• Segment Analysis: Add a segment column to your CSV upload to automatically generate per-segment breakdowns (see "Segment analysis" in the skill definition)
• Custom Metrics: Substitute the primary conversion metric with revenue, retention, or any KPI—the statistical template adapts automatically
• Automation Pipelines: Chain /analyze-test with the sql-queries skill (via /write-query) to generate automated monitoring queries that track your shipped variant's performance over time

Summary

• Install the pm-data-analytics plugin via claude plugin install pm-data-analytics@pm-skills to enable A/B test capabilities
• The ab-test-analysis skill in pm-data-analytics/skills/ab-test-analysis/SKILL.md handles validation, statistical testing, and decision frameworks
• Invoke the /analyze-test command with summary statistics, CSV data, or screenshots to trigger the analysis pipeline
• The plugin automatically checks for sample size adequacy, calculates p-values and confidence intervals, and maps results to ship/extend/stop recommendations
• Generate automated monitoring SQL by combining the A/B test skill with the plugin's SQL generation capability

Frequently Asked Questions

What statistical tests does the pm-data-analytics plugin use for significance testing?

The plugin employs z-tests and chi-squared (χ²) tests to calculate p-values and determine statistical significance, as defined in lines 33-38 of pm-data-analytics/skills/ab-test-analysis/SKILL.md. When raw data is provided, the skill generates Python code to execute these tests programmatically rather than relying on approximations.

Can I analyze raw CSV files with the /analyze-test command?

Yes. The command accepts four input formats including raw CSV uploads (lines 22-27 of pm-data-analytics/commands/analyze-test.md). If you provide user-level data, the skill automatically generates a Python script to calculate conversion rates, confidence intervals, and p-values directly from your dataset.

How does the plugin determine whether to ship or stop an experiment?

The decision framework (lines 49-55 of SKILL.md) maps statistical outcomes and business constraints to four actions: ship (significant positive lift), extend (underpowered but promising), stop (significant negative impact), or investigate (inconclusive results with guardrail violations). This determination considers p-values, practical significance thresholds, minimum detectable effects, and external validity checks.

Do I need Python or R installed on my machine to run the analysis?

No. While the skill generates Python code when processing raw data (line 74), the execution environment is handled by the Claude system. You only need the pm-data-analytics plugin installed via the marketplace. However, if you choose to run the generated scripts locally for custom analysis, Python with standard statistical libraries (pandas, scipy) would be required.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}