π Statistics & Probability
Statistical Significance Calculator
Test if your A/B test result is statistically significant. Free significance calculator with p-value and confidence.
What This Calculator Tells You
Use this calculator to check whether the difference between two percentages is real or could easily be chance. It works for any two groups where you count successes out of a total: email opens in two regions, survey answers from two customer groups, conversion rates in two months, or two versions of a page.
It runs a two-proportion z-test and reports the difference (lift), the z-score and the p-value. A result is flagged significant when p is below 0.05.
What to Enter
- Variant A β visitors: the size of the first group (people emailed, survey respondents, sessions).
- Variant A β conversions: how many in that group did the thing you are measuring (opened, said "yes", bought).
- Variant B β visitors and conversions: the same counts for the second group.
Enter counts, not percentages. Each person should belong to only one group.
Worked Examples
1. Email open rates in two regions
A newsletter went to 2,400 subscribers in the West (528 opened, 22%) and 2,600 in the East (650 opened, 25%).
- Enter 2,400 / 528 and 2,600 / 650 β p = 0.0125 β significant, z = 2.497.
- The East's higher open rate is unlikely to be chance. Worth asking why β send time, subject line, audience?
2. Two customer groups in a survey
12% of 800 new customers (96) and 14% of 750 long-term customers (105) said they would recommend a competitor.
- Enter 800 / 96 and 750 / 105 β p = 0.2415 β not significant.
- A two-point gap with these sample sizes could easily be noise; don't build a strategy on it yet.
3. Conversion rate this month vs last month
September had 12,000 sessions and 360 orders (3.0%); October had 12,500 sessions and 450 orders (3.6%).
- Enter 12,000 / 360 and 12,500 / 450 β p = 0.0086 β significant.
- The rise is real, but this was not a controlled test: seasonality, a sale or a traffic-mix change could explain it as well as your site changes.
How It’s Calculated
- B: Variable B
- z: Z-score (standard normal)
- p: Sample proportion / p-value
- alpha: Significance level
How to Read Your Result
- Significant (p < 0.05): a gap this large would appear less than 5% of the time if the two groups truly had the same rate.
- Not significant: the data cannot separate the groups yet. That is not proof they are equal.
- Significant is not the same as important. With very large samples, even a 0.1-point difference can be significant but too small to matter.
- Running a website experiment? The A/B test calculator page covers test planning, stopping rules and sample-ratio checks.
Common Mistakes to Avoid
- Entering percentages instead of counts. The test needs the raw numbers to judge how much data supports each rate.
- Treating before/after comparisons as experiments. Without random assignment, a significant difference shows that something changed, not what caused it.
- Counting people twice. If the same person can be in both groups, the test's assumptions break.
- Testing many pairs and reporting the one that "worked". Run enough comparisons and some will look significant by chance.
- Using very small groups. If either group has fewer than about 10 successes or 10 failures, treat the result with caution.
When to Use This Calculator
- Compare response or conversion rates between regions, segments or channels.
- Check whether a change in a monthly KPI is more than noise.
- Test survey differences between two customer groups.
- Check the significance of an A/B test result.
Frequently Asked Questions
What test does this calculator use?
A two-tailed two-proportion z-test with a pooled standard error β the standard test for comparing two percentages from independent groups.
What does "statistically significant" mean?
That the observed difference would be unlikely (here, under 5% likely) if the two groups really had the same underlying rate. It does not tell you how large or important the difference is.
What sample size do I need?
It depends on how small a difference you want to detect. Use the sample size calculator before collecting data.
Can I compare averages, like order value, with this?
No β this test is for percentages (counts out of a total). Comparing averages needs a t-test.
Do you store the values I enter?
No. All calculations happen in your browser. Your data never leaves your device.
Related Guides
Related Calculators
Standard Deviation Calculator
Calculate sample and population standard deviation, variance, mean, and range from any list of numbers. Free online tool.
Calculate Now β ΟΒ²Variance Calculator
Calculate sample and population variance from any dataset. Free variance calculator with mean and SD output.
Calculate Now β πZ Score Calculator
Calculate z-score from a value, mean, and standard deviation. Free z-score calculator with probability lookup.
Calculate Now β πP Value Calculator
Calculate p-value from a z-score or t-score. Free p-value calculator for one-tailed and two-tailed tests.
Calculate Now β