Data

Z-Scores and P-Values: A Practical Guide for Analysts

Z-scores and p-values show up everywhere from exam results to A/B tests, and they are often confused. In short: a z-score tells you how unusual a single value is; a p-value tells you how unusual a result is if nothing were really going on. This guide explains both with worked examples.

The z-score

A z-score measures how many standard deviations a value is from the mean:

z = (X − μ) ÷ σ

where X is the value, μ the mean and σ the standard deviation. Positive means above average, negative below.

Example 1: an exam score

A student scores 85; the class mean is 70 and the standard deviation 10. z = (85 − 70) ÷ 10 = 1.5. In a normal distribution, about 93.3% of scores fall below that — roughly the 93rd percentile.

Example 2: spotting an unusual delivery time

Orders usually take 3.9 days to arrive, with a standard deviation of 0.35 days. One shipment takes 4.6 days: z = (4.6 − 3.9) ÷ 0.35 = 2.0. Only about 2.3% of shipments would be this slow if delivery times are roughly normal — worth a look.

Try your own numbers in the z-score calculator.

The 68–95–99.7 rule

• About 68% of values fall within ±1 standard deviation of the mean.
• About 95% fall within ±2.
• About 99.7% fall within ±3.

So |z| above 2 is unusual and above 3 is rare. These percentages assume a bell-shaped (normal) distribution; skewed data like incomes or order values will not follow them closely.

The p-value

A p-value answers: if there were no real effect, how likely is a result at least this extreme? In many tests, you first calculate a z-statistic and then convert it into a p-value.

• z = 1.96 → p = 0.050 (two-tailed)
• z = 2.0 → p ≈ 0.046
• z = 2.1 → p ≈ 0.036

Convert any z-statistic with the p-value calculator.

Example 3: an A/B test

Two checkout pages each get 10,000 visitors. Version A converts 4.0%, version B 4.8%. A two-proportion z-test gives z ≈ 2.76 and p ≈ 0.006. A difference that large would happen well under 1% of the time if the pages truly performed the same, so the result is statistically significant. Run your own test in the A/B test calculator.

One-tailed vs two-tailed

A two-tailed test asks “is there a difference in either direction?” A one-tailed test asks only about one direction, and gives half the p-value for the same z. Use two-tailed unless you decided before collecting data that only one direction matters — switching afterwards to get under 0.05 is a classic way to fool yourself.

What p-values do not tell you

• They are not the probability your hypothesis is true. p = 0.03 does not mean a 97% chance that B is better.
• They do not measure size or importance. With huge samples, trivial differences become “significant”. Always report the effect size, such as the lift.
• 0.05 is a convention, not a cliff. p = 0.049 and p = 0.051 are practically the same evidence.
• Multiple tests inflate false positives. Check 20 metrics and one will probably cross 0.05 by luck.

Z or t?

Z-tests assume you know the population standard deviation or have a large sample. For small samples where the standard deviation is estimated, a t-test is more appropriate; its critical values are slightly larger, especially below about 30 observations.

FAQs

What is a “good” z-score? It depends on context. For test scores, higher is better; for costs or defect rates, a negative z-score may be good news.

Can a p-value be zero? Not exactly, but it can be extremely small. Calculators often show “<0.0001”.

How do z-scores relate to confidence intervals? The z critical value sets the width of the interval — 1.96 for 95% confidence. See confidence intervals and sample size.