π§ͺ A/B Testing & Experiments
A/B Test Calculator
Run a two-proportion z-test on your A/B test results. Free A/B testing calculator with significance and lift.
What This Calculator Tells You
This calculator tells you whether the difference between two versions of a page, email or ad is likely to be real or just random noise. It runs a two-proportion z-test on your visitors and conversions and reports the lift, z-score and p-value.
A result is marked significant when the p-value is below 0.05 β that is, when a difference this large would happen less than 5% of the time if the two versions actually performed the same.
What to Enter
- Variant A β visitors β people who saw the control (your current version).
- Variant A β conversions β how many of them completed the goal: a purchase, sign-up or click.
- Variant B β visitors and conversions β the same counts for the new version.
Count unique visitors who were randomly assigned to each version, and use the same conversion goal for both.
Worked Examples
1. A promising result that is not significant yet
A checkout test sends 5,000 visitors to each version. A converts 200 (4.0%), B converts 240 (4.8%).
- Enter 5,000 / 200 and 5,000 / 240 β 20% lift, z = 1.950, p = 0.0511 β not significant.
- B is probably better, but the evidence falls just short of the 95% standard.
2. The same conversion rates with twice the traffic
The test keeps running until each version has 10,000 visitors: A converts 400, B converts 480.
- Enter 10,000 / 400 and 10,000 / 480 β 20% lift, z = 2.758, p = 0.0058 β significant.
- Identical conversion rates, but more data makes the result trustworthy.
3. A big lift on a small sample
A landing-page test has 200 visitors per version: A converts 10 (5%), B converts 16 (8%).
- Enter 200 / 10 and 200 / 16 β 60% lift, p = 0.2236 β not significant.
- Large lifts on small samples are common and usually shrink as more data arrives.
How It’s Calculated
- z: Z-score (standard normal)
- p: Sample proportion / p-value
How to Read Your Result
- p below 0.05: the difference is statistically significant at 95% confidence. It is unlikely to be pure chance β though not impossible.
- p above 0.05: you cannot yet tell the versions apart. This does not prove they are the same; you may need more traffic.
- Lift is B's conversion rate relative to A's. A significant but tiny lift may not be worth shipping.
- Plan the sample size first. Use the sample size calculator before the test, not after.
Common Mistakes to Avoid
- Stopping the test the moment it turns significant. Checking repeatedly and stopping at the first p < 0.05 inflates false positives. Decide the sample size in advance.
- Testing for too short a period. Run for at least one full week so weekday and weekend behaviour are both included.
- Uneven or broken splits. If you planned 50/50 but got 47/53, check the setup before trusting the result.
- Changing the test mid-way. Editing a variant or the goal resets what you are measuring.
- Testing many metrics and reporting the winner. The more metrics you check, the more likely one looks significant by chance.
When to Use This Calculator
- Decide whether to ship a new landing page, checkout or pricing layout.
- Check results reported by an email or ad platform.
- Explain to stakeholders why a test needs more time.
- Compare conversion rates between two segments or periods.
Frequently Asked Questions
What does a p-value of 0.05 mean?
If the two versions truly converted at the same rate, you would see a difference at least this large about 5% of the time. It is not the probability that B is better.
Is 95% confidence always required?
No β it is a convention. Low-risk changes can use 90%; changes to pricing or checkout often deserve 99%. Choose before the test starts.
How long should I run an A/B test?
Until you reach the sample size you planned, and for at least one full business week. Use the sample size calculator to plan it.
What is the difference between frequentist and Bayesian A/B testing?
This calculator is frequentist: it gives a p-value. Bayesian tests give the probability that B beats A. See our Bayesian A/B test calculator and comparison guide.
Can I test more than two versions?
You can compare each variant with the control, but testing many variants increases the chance of a false winner. Use a stricter threshold or a method designed for multiple comparisons.
Do you store the values I enter?
No. All calculations happen in your browser. Your data never leaves your device.