P-value
A p-value is the probability of seeing a result at least as extreme as yours if there were truly no effect. The smaller it is, the more surprising your result would be without one.
Select any words here to ask your Maestro about them.
A team runs 20 A/B tests in a year. In every one, the change truly makes no difference. They call a win whenever p < 0.05. About how many wins do they report?
What it is: a p-value measures how surprising your data would be if there were no real effect. It shows up in A/B tests, surveys and studies: a small p-value says a gap like yours would rarely turn up in your sample by chance alone if nothing were going on.
What it isn't: it's not the probability that your hypothesis is true, or that the result is "just luck". It doesn't tell you how big the effect is, either: that's what a confidence interval is for.
Where it breaks: when you check a test every day and stop as soon as it looks good, or run many comparisons and report only the ones that came out significant.
How it's used
How to read one
A/B test: a new checkout button
- Version A: 200 orders from 5,000 visitors (4.0%)
- Version B: 230 orders from 5,000 visitors (4.6%)
- marked: p = 0.14If the button made no difference, a gap of 0.6 points or more, in either direction, would show up 14% of the time.
- Not below 0.05: the test can't tell the gap apart from luckIt doesn't prove the button does nothing, either.
When to look closer
A tiny p-value from a huge test
10 million visitors per version
- marked: A: 4.00%, B: 4.03%. p = 0.0006
With 10 million visitors per version, a test gives p = 0.0006. Is the new version a big win?
Ask your Maestro about P-value
It answers from the free course page closest to this word, the way it answers while you study.
You: What does "P-value" mean, in plain words?
Your Maestro: I'll explain it with a case of yours: it sticks better than a definition. Which change at work did you call a win after a single test?
Asking about: Answers from the page "Did it grow more in pesos or in percent? A chart's scale"
Only this browser keeps what you ask here. How it works
Learn more
Related terms
- Confidence interval
A confidence interval is the range where the whole population's number probably falls, worked out from a sample: 52% ± 5 runs from 47% to 57%.
- Population and sample
The population is the whole group you want to know about; the sample is the part of it you actually measure to estimate the rest.
- Probability
Probability measures how likely something is, from 0 (never) to 1 (certain). Favorable outcomes ÷ possible outcomes: rolling a 6 on a die is 1/6.
- Sampling
Sampling is how you pick your sample. The four types: simple random, systematic (one in every so many), stratified (random within each group) and cluster.