Is an extreme value an error or real data?
Decide what each extreme value is by looking at its other columns: impossible, explainable or pending, and write it down before you touch it.
Free course. Your Maestro, an AI, .
You have a list of suspects: three trips of more than 1,300 minutes, six of less than a minute, a 96-minute ride and three people born in 1900. An extreme value is a value far from the rest. That doesn't make it an error. There are real extremes (a bulk order, a long ride) and errors that don't even look extreme.
The answer is almost never in the column where you spotted the extreme. It's in the other columns of the same row: what time it started, how far it went, which station it left from, what type of user it was. A row is a whole case; you read the whole case.
Start with the easiest suspect. Place a bet before you read on.
Check yourself: Trip 40478 lasts 96.4 minutes, five times the median. It started on Saturday, March 14 at 11:07 a.m. at Parque Alameda, ended at the same station and was taken by an occasional user. What is most likely?
With each suspect you ask the same question: what do its other columns say? The answers fall into three families. Sometimes the other columns make the value impossible: a 23-hour trip that covered less than a kilometer. That's a probable error. Sometimes they explain it: the Saturday ride. That's real data, even if it's extreme. And sometimes they aren't enough to decide: you leave it pending and ask whoever knows the data. All three end up in the same place.
A value far from the rest
What do its other columns say?
Path 1 of 3: they make it impossible
Probable error
You set it apart from the figures it affects
Continues at: Write it in your log
Path 2 of 3: they explain it
Real data
It stays in the analysis
Continues at: Write it in your log
Path 3 of 3: they aren't enough
Pending
You ask whoever knows the data
Continues at: Write it in your log
Write it in your log
Three trips of more than 1,300 minutes
Worked example · 5 steps
Three trips last between 1,317 and 1,453 minutes: almost a whole day. Are they errors or real trips?
Work out the next step yourself, then open it to compare.
Try it: the trips under a minute
Now the other extreme. Run the code and read the other columns of the trips under a minute. Before you decide, ask yourself what would have to happen for a trip to last 20 seconds.
Run the code and look at the other columns of the trips under a minute. Then change the filter to < 5 and compare the extra trips that show up.
Run the code and look at the other columns of the trips under a minute. Then change the filter to < 5 and compare the extra trips that show up.
Run your code to see what it prints and the value of its last expression. Every run starts fresh. Ctrl+Enter runs it. Tab indents; to leave the editor, press Escape, then Tab.
Ready to run your code.
Run your code to see its output here.
Data: viajes-rueda, 311 rows
Enter saves a cell and Escape leaves it as it was. Leave a cell empty for a missing value.
| Row | Remove | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | ||||||||||
| 2 | ||||||||||
| 3 | ||||||||||
| 4 | ||||||||||
| 5 | ||||||||||
| 6 | ||||||||||
| 7 | ||||||||||
| 8 | ||||||||||
| 9 | ||||||||||
| 10 | ||||||||||
| 11 | ||||||||||
| 12 | ||||||||||
| 13 | ||||||||||
| 14 | ||||||||||
| 15 | ||||||||||
| 16 | ||||||||||
| 17 | ||||||||||
| 18 | ||||||||||
| 19 | ||||||||||
| 20 |
The six trips under a minute start and end at the same station. Most likely a false start: someone takes the bike out, doesn't like something (the seat, a tire) and puts it back. It's real, it happened, but it isn't a trip. With the < 5 filter two more show up, of 4.4 and 4.9 minutes, from Jardín Juárez to Plaza de Armas, half a kilometer: short, normal trips. The cutoff matters.
Here's your list of suspects now:
| Suspect | What the other columns say | What it is | What you do |
|---|---|---|---|
| 3 trips of more than 1,300 min | under 1 km, start at night, end the next day around 8:40 p.m. | Probable duration error | Out of the duration figures; still counted as trips; ask |
| 6 trips of less than 1 min | same station, 0 km | Real, but not trips | Out of the duration figures; write down the cutoff |
| Trip 40478, 96.4 min | Saturday 11:07 a.m., occasional, out and back | Real data | It stays |
| 3 birth years of 1900 | three subscribers, round year | Probable filler value | Pending; don't use it as an age |
None of this changes the table yet. Removing, correcting or flagging rows is cleaning, and another free Maestro course covers that. If you're unsure about a case, tell your Maestro along with the other columns of the row. And if you want to practice the judgment call, ask for a new case.