A/B testing
A/B testing is comparing two versions of something, like a web page or an email, by showing each one to a random half of people and measuring which does better.
Select any words here to ask your Maestro about them.
A new checkout truly lifts conversion from 3.0% to 3.6%. You test it with 2,000 visitors per version. If 20 teams ran this same test, how many would see a significant win?
How it works: you split people at random: half see version A (what you have now) and half see B (the change). You measure the same number for both, like conversion rate, and a test like the chi-square test or the t-test tells you whether the gap is bigger than chance, with its p-value.
What it's for: deciding with data instead of opinions: which subject line gets opened, which page sells more, which price converts better. Because the split is random, the only systematic difference between the groups is the change.
Where it breaks: checking results every day and stopping the first time B is ahead. Early on, the numbers swing a lot, so stopping at a good moment turns luck into "wins". Decide how many visitors you need before you start, and stop there.
How it's used
How to run one
An A/B test, step by step
- Pick one number to judge it by: here, the share of visitors who buy
- marked: Work out how many visitors you need with a sample size calculatorIt depends on your current rate and the smallest lift worth finding.
- Split visitors at random, 50/50, until you reach that number
- Compare the two rates and their p-value, then decide
When to look closer
Counting the visitors first
Two weeks of testing
- You split traffic 50/50 for two weeks
- marked: A got 10,000 visitors and B got 9,400
You split traffic 50/50, but A got 10,000 visitors and B got 9,400. Can you still trust the test?
Ask your Maestro about A/B testing
It answers from the free course page closest to this word, the way it answers while you study.
You: What does "A/B testing" mean, in plain words?
Your Maestro: I'll explain it with a case of yours: it sticks better than a definition. Which change to your site or emails would you like to test?
Asking about: Answers from the page "How do I compare each row with the previous month in SQL with LAG?"
Only this browser keeps what you ask here. How it works
Learn more
Related terms
- P-value
A p-value is the probability of seeing a result at least as extreme as yours if there were truly no effect. The smaller it is, the more surprising your result would be without one.
- t-test
A t-test checks whether the difference between two averages, like the average basket before and after a change, is likely real or could be chance.
- Chi-square test
A chi-square test checks whether two categories are related, like plan type and churn, by comparing the counts you got with the counts you'd expect if they weren't.
- Population and sample
The population is the whole group you want to know about; the sample is the part of it you actually measure to estimate the rest.