Skip to content
Courses

Simpson's paradox

Simpson's paradox is when a trend that shows up in every group reverses once the groups are combined, because each side has a different mix of groups.

Select any words here to ask your Maestro about them.

The new checkout converts better on phones (2.4% vs 2.0%) and on computers (6.6% vs 6.0%), but 80% of its visitors were on phones, against 30% of the old one's. The old one converts 4.8% overall. The new one converts about:

The old checkout's visitors in March against the new one's in April, after a phone ad campaign; 10,000 visitors each. Simulated data.

Why it happens: each version's overall rate is a weighted average of its groups, so it depends on how many people each version had in each group.

Where it shows up: in A/B tests with uneven traffic, in treatment success rates, in admissions and in any pivot table that shows only the total.

What to do: before comparing totals, split by the variable that shifts the mix and check whether the conclusion holds. Then ask which one answers your question: it isn't always the breakdown.

How it's used

Berkeley admissions

Berkeley, 1973

  1. Men admitted: 44% of 8,442 applications
  2. Women admitted: 35% of 4,321 applications
  3. marked: Department by department, women were admitted about as often as men, often moreWomen applied more to the departments that admitted a smaller share of everyone who applied.
The total looked like bias against women; the groups showed where each person applied. Splitting by the right variable changed the conclusion.

When to look closer

Two raises and one average

Average yearly pay

  1. Engineering: 60 people, from $150,000 to $154,500 (+3%)
  2. Support: from 40 to 140 people, from $60,000 to $61,800 (+3%)

Engineering and support both got a 3% raise this year. Can the company's average pay have gone down?

Ask your Maestro about Simpson's paradox

It answers from the free course page closest to this word, the way it answers while you study.

Example

You: What does "Simpson's paradox" mean, in plain words?

Your Maestro: I'll explain it with a case of yours: it sticks better than a definition. Which total do you check that might flip if you split it by group?

Asking about: Answers from the page "How do I show each group's total next to every row in SQL?"

Only this browser keeps what you ask here. How it works

Learn more

Related terms

  • Weighted average

    A weighted average gives each value a weight for how much it counts, instead of treating them all the same: multiply each value by its weight, add them up and divide by the sum of the weights.

  • A/B testing

    A/B testing is comparing two versions of something, like a web page or an email, by showing each one to a random half of people and measuring which does better.

  • Pivot table

    A pivot table summarizes a long table in seconds: it groups by the columns you pick and sums, counts or averages another, with no formulas.

  • Correlation vs. causation

    Correlation means two numbers move together; causation means changing one changes the other. Moving together doesn't prove that one causes the other.

The whole glossary