How do I count the missing values in each column with pandas?

The third step of the method has two halves: how many values are missing and why. This page is the first half. On the previous page, data.info() already left you a clue: duracion_min, estacion_destino and distancia_km have 303 values each, so each one is missing exactly 8. Before counting anything, place a bet.

Check yourself: Three columns have exactly 8 missing values each: duracion_min, estacion_destino and distancia_km. What do you think is going on?

Counting the gaps

data.isna() returns a table the same size as data, with True where a value is missing and False where it isn't. Summing it counts the True values in each column, because Python treats True as 1. And since the average of a set of ones and zeros is the share of ones, isna().mean() gives you the fraction of gaps.

Run the cell to count the gaps in each column. Then change it to (data.isna().mean() * 100).round(1) to see them as percentages. You're done when you know which column has the highest share of gaps and how high it is.

Your code (Python)

Run your code to see what it prints and the value of its last expression. Every run starts fresh. Ctrl+Enter runs it. Tab indents; to leave the editor, press Escape, then Tab.

Ready to run your code.

Run your code to see its output here.

Data: viajes-rueda, 311 rows

Enter saves a cell and Escape leaves it as it was. Leave a cell empty for a missing value.

The exercise's data, "viajes-rueda". Your edits run with your code.
Row										Remove
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

Page 1 of 16

Run it yourself

Six columns are complete. Three are missing 2.6% (8 of 311) and anio_nacimiento is missing 23.5% (73 of 311), almost one row in four.

A percentage helps you size things up, but it doesn't decide anything on its own. A 2.6% concentrated on the last day can ruin that day's report; a 23.5% concentrated in one group can wipe out that whole group if you drop the rows. That's why the next question is where the gaps are, and the first way to find out is to see whether several columns go empty together.

First, practice on a small table. Write down what each line prints and why.

Gaps that go together

example.py

import pandas as pd

viajes = pd.DataFrame({
    "duracion_min": [12.5, None, 30.0, None, 8.4],
    "estacion_destino": ["Centro", None, "Estadio", None, "Mercado"],
    "anio_nacimiento": [1990, 1985, None, None, 2001],
})

faltan_ambas = viajes["duracion_min"].isna() & viajes["estacion_destino"].isna()
print(viajes.isna().sum().sum())
print(faltan_ambas.sum())
print(viajes.isna().any(axis=1).sum())

What three numbers does it print?

Why?

A guess is fine: it's kept as you wrote it, so you can compare. Only you and your Maestro see your predictions.

Make your prediction

Now on the Rueda table. The code saves two masks, one per column, and counts the rows where both are true.

Run the cell. Then add a third mask for distancia_km and count the rows where all three are missing at once. Finish by showing those rows with data[faltan_duracion]. You're done when you can see the 8 rows and what they have in common.

Your code (Python)

Run your code to see what it prints and the value of its last expression. Every run starts fresh. Ctrl+Enter runs it. Tab indents; to leave the editor, press Escape, then Tab.

Ready to run your code.

Run your code to see its output here.

Data: viajes-rueda, 311 rows

Enter saves a cell and Escape leaves it as it was. Leave a cell empty for a missing value.

The exercise's data, "viajes-rueda". Your edits run with your code.
Row										Remove
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

Page 1 of 16

Run it yourself

They're the same 8 trips: they have no destination station, and so no distance (it's computed between two stations) and no duration (it's measured until the bike is docked). Three empty columns, one cause. When several gaps go together, there's almost always a single fact behind them, and finding it is more useful than treating each column separately. Look at the inicio column of those 8 rows: the next page starts there.

Gaps that don't look like gaps

isna() counts only what pandas recognizes as missing (its documentation notes that an empty string, '', doesn't count either). In real-world tables, missing data shows up in many other forms:

text that says it's missing: "N/A", "sin dato" ("no data"), "-", a cell with a single space;
a filler value the system puts in when there's no data: 0, 9999, 1900.

Rueda has one of these. In anio_nacimiento there are three rows with 1900, a birth year no bike-share rider could plausibly have; isna() doesn't count them. You'll see them in module 3. If the filler text comes from the file itself, you can tell pandas about it when you read the file; another free Maestro course, on CSV files that open wrong, covers that.

Your log entry for this page could be:

Observation: 8 trips have no destination, distance or duration; 73 are missing the birth year. Evidence: isna().sum() and the combined count of the three masks. Question: why do those 8 trips have no destination? And why is almost one person in four missing a birth year?

If the gaps in your table are written as text, ask your Maestro how to find them.

Try it

Check yourself: In a new table, isna().sum() answers 0 for the column telefono. Can you be sure no phone number is missing?

pandas: DataFrame.isna, in the official documentation