# How PostHog Experiments Statistical Analysis Calculates Significance: A Bayesian Deep Dive

> Discover how PostHog calculates experiment significance using Bayesian methods and win probability. Learn about posterior sampling for count, continuous, and funnel metrics. Skip p-values.

- Repository: [PostHog/posthog](https://github.com/PostHog/posthog)
- Tags: deep-dive
- Published: 2026-04-25

---

**PostHog determines experiment significance using a Bayesian approach that combines win-probability calculations with expected-loss thresholds, evaluating count-based, continuous, and funnel metrics through posterior sampling rather than traditional p-values.**

PostHog's experiments statistical analysis calculates significance using a sophisticated Bayesian framework implemented across three specialized modules in the PostHog/posthog repository. Unlike frequentist A/B testing that relies on p-values, the system employs probabilistic modeling to quantify both the likelihood that a variant is best and the potential cost of choosing incorrectly. This methodology provides product teams with intuitive, risk-aware significance determinations for any metric type.

## The Bayesian Significance Framework

The PostHog experiments engine processes three distinct metric families, each with specialized statistical models:

### Count and Rate Metrics

For events-per-exposure calculations (e.g., "how many purchases per user"), PostHog uses a **Gamma-Poisson** conjugate prior model. The implementation in [`posthog/hogql_queries/experiments/trends_statistics_v2_count.py`](https://github.com/PostHog/posthog/blob/main/posthog/hogql_queries/experiments/trends_statistics_v2_count.py) treats each variant's event rate as a Gamma-distributed random variable, allowing for posterior updating that accounts for both the count of events and exposure volume.

### Continuous Metrics

For revenue, duration, or other unbounded continuous values, the system employs a **Normal-Inverse-Gamma** posterior on the log-mean, effectively modeling the data as log-normal. The source in [`posthog/hogql_queries/experiments/trends_statistics_v2_continuous.py`](https://github.com/PostHog/posthog/blob/main/posthog/hogql_queries/experiments/trends_statistics_v2_continuous.py) handles skewed distributions common in monetary metrics by working in log-space before transforming back to the original scale.

### Funnel Conversion Metrics

For conversion-rate experiments, PostHog implements a **Beta-Binomial** model in [`posthog/hogql_queries/experiments/funnels_statistics_v2.py`](https://github.com/PostHog/posthog/blob/main/posthog/hogql_queries/experiments/funnels_statistics_v2.py). This approach treats conversion rates as Beta-distributed probabilities updated by binomial observations (successes vs. failures).

## The 7-Step Significance Calculation Process

Regardless of metric type, the experiments statistical analysis follows a rigorous seven-step pipeline to calculate significance:

### 1. Exposure Validation

Before statistical evaluation, each variant must satisfy minimum sample size requirements. The system checks against `FF_DISTRIBUTION_THRESHOLD` (defined in the experiment configuration) in the `are_results_significant_*` functions:

- **Count metrics**: Validated in [`trends_statistics_v2_count.py`](https://github.com/PostHog/posthog/blob/main/trends_statistics_v2_count.py) lines 136–143
- **Continuous metrics**: Validated in [`trends_statistics_v2_continuous.py`](https://github.com/PostHog/posthog/blob/main/trends_statistics_v2_continuous.py) lines 58–66  
- **Funnel metrics**: Validated in [`funnels_statistics_v2.py`](https://github.com/PostHog/posthog/blob/main/funnels_statistics_v2.py) lines 78–82

If any variant falls below the threshold, the function returns `ExperimentSignificanceCode.NOT_ENOUGH_EXPOSURE`.

### 2. Posterior Sampling and Win Probabilities

The engine draws `SAMPLE_SIZE` samples from each variant's posterior distribution to estimate the probability that each variant is the best:

- **Count**: Uses `gamma.rvs` to draw from the Gamma posterior (`calculate_probabilities_v2_count`, lines 71–93)
- **Continuous**: Draws from the t-distribution posterior in log-space (`calculate_probabilities_v2_continuous`, lines 71–93)  
- **Funnel**: Uses `betabinom.rvs` for Beta-Binomial sampling (`calculate_probabilities_v2`, lines 24–48)

The resulting `probabilities` array contains the fraction of draws where each variant outperforms all others.

### 3. Probability Threshold Check

If the highest win probability is **≥ 0.9** (`MIN_PROBABILITY_FOR_SIGNIFICANCE`), the experiment passes the first significance gate. Otherwise, `are_results_significant_*` returns `LOW_WIN_PROBABILITY` (count: lines 146–152, continuous: lines 69–71, funnel: lines 84–88).

### 4. Best Variant Identification

The system identifies the "best" variant based on empirical rates:
- **Count**: Highest count/exposure ratio
- **Continuous**: Highest raw mean
- **Funnel**: Highest conversion rate (success_count / (success_count + failure_count))

### 5. Expected Loss Calculation

PostHog calculates the **expected loss**—the average penalty incurred if the chosen best variant is actually suboptimal:

- **Count**: `calculate_expected_loss_v2_count` (lines 56–71) draws from the best variant's posterior and competitors' posteriors, computing the mean positive difference
- **Continuous**: `calculate_expected_loss_v2_continuous` (lines 85–106) performs the calculation in log-space using the t-distribution posterior
- **Funnel**: `calculate_expected_loss_v2` (lines 20–38) averages the surplus of competitors' conversion rates over the target's

This produces a loss value between 0 and 1 (or monetary units for continuous metrics).

### 6. Loss Threshold Application

If the expected loss exceeds **0.05** (`EXPECTED_LOSS_SIGNIFICANCE_LEVEL`), the result is `HIGH_LOSS`. Otherwise, the experiment achieves `SIGNIFICANT` status:

- **Count**: Threshold check at lines 58–63 in [`trends_statistics_v2_count.py`](https://github.com/PostHog/posthog/blob/main/trends_statistics_v2_count.py)
- **Continuous**: Lines 78–84 in [`trends_statistics_v2_continuous.py`](https://github.com/PostHog/posthog/blob/main/trends_statistics_v2_continuous.py)
- **Funnel**: Lines 92–98 in [`funnels_statistics_v2.py`](https://github.com/PostHog/posthog/blob/main/funnels_statistics_v2.py)

### 7. Significance Code Return

The final output is an `ExperimentSignificanceCode` enum value (defined in [`posthog/schema.py`](https://github.com/PostHog/posthog/blob/main/posthog/schema.py)), which the frontend consumes to display status badges like "Significant" or "High Loss".

## Implementation Examples by Metric Type

### Count-Based Analysis (Events per Exposure)

```python
from posthog.schema import ExperimentVariantTrendsBaseStats
from posthog.hogql_queries.experiments.trends_statistics_v2_count import (
    calculate_probabilities_v2_count,
    are_results_significant_v2_count,
)

control = ExperimentVariantTrendsBaseStats(
    key="control", count=120, exposure=1, absolute_exposure=5000
)
test_a = ExperimentVariantTrendsBaseStats(
    key="test_a", count=150, exposure=1, absolute_exposure=5000
)

# Calculate win probabilities via Gamma posterior sampling

probabilities = calculate_probabilities_v2_count(control, [test_a])

# Returns: [0.12, 0.88] (control wins 12%, test_a wins 88%)

# Determine significance with expected loss check

code, loss = are_results_significant_v2_count(control, [test_a], probabilities)

# Returns: (ExperimentSignificanceCode.SIGNIFICANT, 0.03)

```

This example demonstrates the Gamma-Poisson model in [`trends_statistics_v2_count.py`](https://github.com/PostHog/posthog/blob/main/trends_statistics_v2_count.py), where `count` represents total events and `absolute_exposure` represents the denominator for rate calculation.

### Continuous Metric Analysis (Revenue)

```python
from posthog.schema import ExperimentVariantTrendsBaseStats
from posthog.hogql_queries.experiments.trends_statistics_v2_continuous import (
    calculate_probabilities_v2_continuous,
    are_results_significant_v2_continuous,
)

control = ExperimentVariantTrendsBaseStats(
    key="control", count=5000.0, exposure=1, absolute_exposure=2000
)  # count stores total revenue

test = ExperimentVariantTrendsBaseStats(
    key="test", count=5600.0, exposure=1, absolute_exposure=2000
)

# Log-normal posterior sampling

prob = calculate_probabilities_v2_continuous(control, [test])
code, loss = are_results_significant_v2_continuous(control, [test], prob)

# Returns: (SIGNIFICANT, 0.07) 

# Interpretation: $0.07 expected loss per user if wrong variant chosen

```

The [`trends_statistics_v2_continuous.py`](https://github.com/PostHog/posthog/blob/main/trends_statistics_v2_continuous.py) implementation treats `count` as the sum of continuous values (like total revenue), modeling the per-user average via Normal-Inverse-Gamma conjugacy.

### Funnel Conversion Analysis

```python
from posthog.schema import ExperimentVariantFunnelsBaseStats
from posthog.hogql_queries.experiments.funnels_statistics_v2 import (
    calculate_probabilities_v2,
    are_results_significant_v2,
)

control = ExperimentVariantFunnelsBaseStats(
    key="control", success_count=200, failure_count=800
)
test = ExperimentVariantFunnelsBaseStats(
    key="test", success_count=260, failure_count=740
)

# Beta-Binomial posterior sampling

prob = calculate_probabilities_v2(control, [test])
code, loss = are_results_significant_v2(control, [test], prob)

# Returns: (SIGNIFICANT, 0.018)

# Interpretation: 1.8% conversion point expected loss if wrong

```

As implemented in [`funnels_statistics_v2.py`](https://github.com/PostHog/posthog/blob/main/funnels_statistics_v2.py), the Beta-Binomial model directly models uncertainty in conversion rates using success and failure counts.

## Summary

- **PostHog employs a Bayesian + expected-loss framework** that avoids p-value limitations by calculating both the probability a variant is best and the cost of potential errors.
- **Three specialized modules** handle distinct metric types: Gamma-Poisson for counts ([`trends_statistics_v2_count.py`](https://github.com/PostHog/posthog/blob/main/trends_statistics_v2_count.py)), Normal-Inverse-Gamma for continuous values ([`trends_statistics_v2_continuous.py`](https://github.com/PostHog/posthog/blob/main/trends_statistics_v2_continuous.py)), and Beta-Binomial for conversions ([`funnels_statistics_v2.py`](https://github.com/PostHog/posthog/blob/main/funnels_statistics_v2.py)).
- **Significance requires dual thresholds**: a win probability ≥ 0.9 (`MIN_PROBABILITY_FOR_SIGNIFICANCE`) and an expected loss ≤ 0.05 (`EXPECTED_LOSS_SIGNIFICANCE_LEVEL`).
- **Exposure validation** occurs before statistical testing, ensuring variants meet minimum sample size (`FF_DISTRIBUTION_THRESHOLD`) to prevent premature conclusions.
- **Results are deterministic** rather than probabilistic—each function returns a concrete `ExperimentSignificanceCode` enum value consumed by the PostHog UI.

## Frequently Asked Questions

### How does PostHog's Bayesian approach differ from traditional frequentist A/B testing?

Traditional frequentist testing relies on p-values and null hypothesis significance testing, which can lead to misinterpretation of "significance" as the probability of a variant being best. PostHog's Bayesian experiments statistical analysis calculates significance by directly computing the **probability that each variant is the best** (win probability) and the **expected loss** from choosing the wrong variant. This provides intuitive, actionable metrics: you know exactly how confident you should be and what it costs to be wrong, rather than just rejecting a null hypothesis.

### What happens if my experiment doesn't meet the sample size requirements?

If any variant falls below `FF_DISTRIBUTION_THRESHOLD` exposures (or sample size for funnels), the system immediately returns `ExperimentSignificanceCode.NOT_ENOUGH_EXPOSURE` from the `are_results_significant_*` functions without performing posterior sampling. This safeguard prevents false positives from insufficient data and is enforced in [`trends_statistics_v2_count.py`](https://github.com/PostHog/posthog/blob/main/trends_statistics_v2_count.py) (lines 136–143), [`trends_statistics_v2_continuous.py`](https://github.com/PostHog/posthog/blob/main/trends_statistics_v2_continuous.py) (lines 58–66), and [`funnels_statistics_v2.py`](https://github.com/PostHog/posthog/blob/main/funnels_statistics_v2.py) (lines 78–82).

### Why does PostHog use expected loss instead of just win probability?

Win probability alone doesn't capture the **magnitude of potential error**. A variant might have a 95% chance of being best, but if the alternative is only 0.01% worse, the risk is minimal. Conversely, a 90% win probability with a 50% performance downside represents significant risk. By calculating expected loss in `calculate_expected_loss_v2_count`, `calculate_expected_loss_v2_continuous`, and `calculate_expected_loss_v2`, PostHog ensures experiments are only marked significant when both the confidence is high (≥90%) and the potential downside is low (≤5% of the metric value).

### Can I adjust the significance thresholds for my experiments?

The thresholds `MIN_PROBABILITY_FOR_SIGNIFICANCE` (default 0.9) and `EXPECTED_LOSS_SIGNIFICANCE_LEVEL` (default 0.05) are constants defined in the statistical modules. While these defaults are hardcoded in the current implementation within `posthog/hogql_queries/experiments/`, they represent industry-standard Bayesian decision theory thresholds that balance discovery speed with decision safety. The `ExperimentSignificanceCode` enum in [`posthog/schema.py`](https://github.com/PostHog/posthog/blob/main/posthog/schema.py) provides the standardized result codes that the frontend interprets to display significance states.