π§ͺ A/B Testing & Experiments
Bayesian A/B Test Calculator
Run a Bayesian A/B test with credible intervals and probability variant beats control. Free Bayesian A/B calculator.
What This Calculator Tells You
A Bayesian A/B test answers the question most teams actually want answered: what is the probability that version B is better than version A? Instead of a p-value, you get a direct probability you can act on.
Enter visitors and conversions for each variant. The calculator simulates thousands of plausible conversion rates for each version and reports how often B beats A, along with each conversion rate and the lift.
What to Enter
- Variant A β visitors / conversions β your control: the number of users who saw it and how many converted.
- Variant B β visitors / conversions β the challenger.
- Use unique users (not page views) and count each user's conversion once. The calculator uses a neutral (uniform) prior, so results are driven by your data.
Worked Examples
1. A clear winner
A: 5,000 visitors, 150 conversions (3.0%). B: 5,000 visitors, 180 conversions (3.6%).
- P(B beats A) β 95%, lift +20%. Most teams would ship B.
2. Promising but not yet convincing
A: 1,200 visitors, 48 conversions (4.0%). B: 1,150 visitors, 52 conversions (4.52%).
- P(B beats A) β 73%, lift +13%. There is still roughly a one-in-four chance A is better β keep the test running.
3. The challenger is losing
A: 800 visitors, 40 conversions (5.0%). B: 800 visitors, 32 conversions (4.0%).
- P(B beats A) β 17%. Stop B unless it has another benefit.
How It’s Calculated
- P: Principal amount
- B: Variable B
How to Read Your Result
- Above 95%: strong evidence B is better.
- 80β95%: likely better; ship if the change is cheap and easy to reverse.
- 20β80%: inconclusive β collect more data.
- Below 20%: B is probably worse.
- Because the calculator uses random simulation, the probability can move by about a percentage point between runs. That is normal.
Common Mistakes to Avoid
- Stopping on the first day B looks good. Run full business cycles (at least one or two weeks) to avoid weekday effects.
- Ignoring effect size. 96% probability of a 0.5% lift may not be worth an engineering change.
- Testing on tiny samples. With a few dozen conversions, probabilities swing wildly.
- Counting sessions instead of users, which double-counts returning visitors.
When to Use This Calculator
- Decide whether to ship a landing page, pricing or email variant.
- Explain test results to stakeholders as a plain probability.
- Monitor a test as data comes in without p-value peeking problems.
- Make faster calls on low-risk, easily reversible changes.
Frequently Asked Questions
What does 'probability B beats A' mean?
Given the data and a neutral starting assumption, it is the chance that B's true conversion rate is higher than A's. 90% means B is better in 9 out of 10 plausible scenarios.
Bayesian or frequentist β which is better?
Both are valid. Bayesian results are easier to explain and act on; frequentist tests give a fixed error rate if you set the sample size in advance. See our A/B test calculator for the frequentist version.
What prior does this calculator use?
A uniform Beta(1,1) prior for each variant, which adds the equivalent of one conversion and one non-conversion. With realistic sample sizes it has almost no influence.
How many visitors do I need?
Enough for at least a few hundred conversions per variant for small lifts. Use the sample size calculator to plan.
Do you store the values I enter?
No. All calculations happen in your browser. Your data never leaves your device.