Data

A/B Test Statistical Significance: What It Really Means (And When to Stop a Test)

“Statistically significant” is the phrase that ends most A/B test discussions — and it is widely misunderstood. A significant result does not mean the new version is definitely better, that the lift you measured is the lift you will get, or that the change matters to the business. It means something narrower and more useful once you understand it.

This guide explains what significance really tells you, how to set up a test so the answer is trustworthy, and when it is safe to stop.

What a p-value actually says

An A/B test compares a control (A) with a variant (B). The p-value answers this question: if there were truly no difference between A and B, how likely is it that we would see a gap at least this big just from random chance?

If that probability is below a threshold you set in advance — usually 5% (p < 0.05) — you call the result statistically significant. You are saying: “random noise alone would rarely produce this, so we will act as if the difference is real.”

What it does not say:

A worked example

A landing page test runs for two weeks:

That is a 10% relative lift, which sounds great. But a two-proportion z-test gives z ≈ 1.22 and p ≈ 0.22. A gap this size would appear by chance about one time in five even if the pages were identical. Not significant.

If B had produced 360 sign-ups (3.6%, a 20% lift), z ≈ 2.38 and p ≈ 0.018 — significant at the 5% level. You can check your own numbers in the A/B test calculator.

Why sample size matters so much

Small lifts need large samples to detect. Detecting a 10% relative lift on a 3% baseline at 95% confidence with 80% power needs roughly 53,000 visitors per variant. A 20% lift on the same baseline needs about 14,000 per variant.

That is why so many tests on low-traffic sites end “inconclusive” — they were never big enough to detect a realistic effect. Before starting, plan the sample with the sample size calculator and ask whether you can reach it in a reasonable time. If not, test bolder changes that could produce bigger effects.

The peeking problem: when to stop a test

The most expensive mistake in testing is stopping the moment the dashboard shows p < 0.05. P-values bounce around while data comes in. If you check every day and stop the first time it dips below 0.05, your real false-positive rate can be several times higher than 5%.

Safer rules:

  1. Decide the sample size before you start, and analyze when you reach it.
  2. Run for whole weeks. Weekday and weekend visitors behave differently; stopping on a Tuesday can bias results.
  3. Run at least one full business cycle — commonly one to two weeks, longer for B2B with weekly patterns.
  4. If you must monitor continuously, use a method designed for it, such as sequential testing or a Bayesian approach with a pre-agreed decision rule.

Significance vs practical importance

With enough traffic, tiny differences become significant. A 0.5% relative lift on a million visitors can have p < 0.01 and still not justify the engineering time to maintain a new variant. Always look at the confidence interval for the lift and ask whether even the low end would be worth it.

The reverse also holds: “not significant” does not prove there is no effect. It may mean the test was too small. The interval tells you how big an effect you can rule out.

Other things that break A/B tests

A simple pre-launch checklist

  1. Write the hypothesis and the primary metric.
  2. Set the minimum lift worth detecting and calculate the sample size.
  3. Choose the significance level (usually 5%) and power (usually 80%).
  4. Confirm tracking works for both variants.
  5. Commit to the stopping rule before you see any data.

Key takeaways