If the alert is 90% accurate, how sure can you be it's fraud?
Turn percentages into cases out of 20,000 to see how many alerts are real when what you're looking for is rare.
Free course. Your Maestro, an AI, .
La Ceiba installed a system at its registers that flags card purchases that look like fraud. The vendor pitches it like this: "Catches 90 out of every 100 frauds." The fine print says it also wrongly flags 5 out of every 100 normal purchases. And the security team knows that, in their stores, roughly 1 in every 200 card purchases is fraud. (The numbers are invented for the course.)
One Tuesday, the register at Plaza Norte lights up red: the system flagged a purchase. The manager has to decide whether to call security and hold the customer.
Check yourself: The system flagged a purchase. How likely is it that it really is fraud?
Think in purchases, not percentages
The percentages in this story have three different denominators, which is why they're confusing: the 90% is of the frauds, the 5% is of the normal purchases and the "1 in 200" is of all purchases. The safest way not to get lost is to drop the percentages and count purchases. Picture a round number of purchases and split them up:
20,000 purchases at the registers
Worked example · 5 steps
1 in 200 purchases is fraud. The system flags 90 out of 100 frauds and 5 out of 100 normal purchases. Out of 20,000 purchases, how many alerts will there be, and how many will be real fraud?
Work out the next step yourself, then open it to compare.
Why it happens: the rare drowns in the common
A 5% error rate looks small, but it applies to a huge number of normal purchases. A 90% hit rate looks big, but it applies to very few frauds. When what you're looking for is rare, false alarms outnumber true ones, even if the system is good.
If you still feel the answer should be 90%, tell your Maestro why and work through it together.
The same happens with anything that looks for something uncommon among many cases: a filter that flags duplicate invoices, a review that looks for data entry errors, a test that detects a rare condition. The question to ask is always the same: out of every 100 alerts, how many are real? And to answer it, you need to know how common the thing you're looking for is.
If your work has alerts, filters or automated reviews, ask your Maestro how many of their alerts you'd expect to be real.
Try it
The code below does the math for the 20,000 purchases. The first four lines are the assumptions; the rest is the calculation you just did by hand. Run it as is, then change the assumptions one at a time (the code keeps its Spanish names and output labels):
- A store where fraud is much more common: change
fraude_de_cada = 200tofraude_de_cada = 20and run it. - Go back to 200 and improve the system: change
error = 0.05toerror = 0.01and run it.
Run the code as it is (8.3 out of every 100 alerts are fraud). Then change fraudedecada to 20 and run it again: you should see 48.6. Then set fraudedecada back to 200, change error to 0.01, and you should see 31.1. Which change moved the result more?
Run the code as it is (8.3 out of every 100 alerts are fraud). Then change fraude_de_cada to 20 and run it again: you should see 48.6. Then set fraude_de_cada back to 200, change error to 0.01, and you should see 31.1. Which change moved the result more?
Run your code to see what it prints and the value of its last expression. Every run starts fresh. Ctrl+Enter runs it. Tab indents; to leave the editor, press Escape, then Tab.
Ready to run your code.
Run your code to see its output here.
What moves the result most isn't how good the system is, but how common fraud is. When 1 in 20 purchases is fraud, almost half the alerts are real. When it's 1 in 200, fewer than one in ten. That's why the same system can be very useful in one place and maddening in another.
In practice, an alert is a reason to check, not proof. The manager at Plaza Norte shouldn't hold anyone: they should ask for ID, as they would with any unusual purchase.