How do I know if my cleaning lost rows? Count before and after every step

So far each fix came with its own check: counting blanks after the map, counting blanks after converting, counting flags. This page pulls it all into a habit that will stay with you for the rest of the course: count the rows before and after every step, and write it down.

The reason is simple. Cleaning steps that lose data almost never raise an error. An incomplete map leaves blanks without warning; a filter takes rows without warning; a dropna() takes everything that has a blank in any column, without warning. The only thing that warns you is a number that changed when it shouldn't have.

Start by predicting what dropna() does on a small table.

dropna() with nothing else

Example 1 of 2

example.py

import pandas as pd

t = pd.DataFrame({
    "folio": ["A", "B", "C", "D", "E"],
    "kilos": [40, None, 25, 30, 12],
    "descuento": [None, 5, None, 10, 10],
})

print(len(t.dropna()))

The table has 5 sales. How many are left after dropna()?

Why?

A guess is fine: it's kept as you wrote it, so you can compare. Only you and your Maestro see your predictions.

Example 2 of 2, opens once you explain this one

Make your prediction

A ten-second function

Nothing fancy is needed. A list and a function that adds one entry each time you finish a step:

Python

bitacora = []

def anota(paso, antes, despues, nota):
    bitacora.append({"paso": paso, "antes": antes, "despues": despues, "nota": nota})
    print(f"{paso:<20} {antes:>3} -> {despues:>3}   {nota}")

Each step records its name, the rows before, the rows after and a note with the why.

After each cleaning step you call anota with the len(ventas) from before and after, and a short note: what it did and why. The function prints one line per step, so the script's output is already a summary you can read at a glance, and the bitacora list can be saved as a CSV next to the clean table (you do that in the last module).

When you read the log, compare each line with what the step was supposed to do:

A step that changes text or types (normalizing the city, converting the amount, flagging impossible values) should leave the same number of rows. If it drops, something got lost.
A step that removes rows on purpose (removing duplicates, keeping the sales that have an amount) should drop exactly what you expected, and the note says how many and why.
A big drop with no explanation is the clearest alarm you'll ever get.

The note matters as much as the numbers. Three months from now, "58 sales" says nothing. "58 left: without V-0036, waiting on its invoice, and without V-0050, which said 'pendiente'" answers the question before anyone asks it.

Don't know what to write in a step's note? Ask your Maestro to help you word it.

Try it

Marisol asks you for a table of collected sales to add up the total: only the sales that have an amount. Someone already wrote the code, with its log, and used dropna(). Run it and read the log. If the drop doesn't add up, ask your Maestro which column is causing it.

Run the code and read the log: the last step was supposed to remove only the sales with no amount. Find out why it takes so many and fix it so it removes only those. You're done when the log says 60 -> 58 on that step.

Your code (Python)

import pandas as pd

ventas = data

bitacora = []

def anota(paso, antes, despues, nota):

bitacora.append({"paso": paso, "antes": antes, "despues": despues, "nota": nota})

print(f"{paso:<20} {antes:>3} -> {despues:>3} {nota}")

anota("leer", len(ventas), len(ventas), "hoja de marzo")

antes = len(ventas)

texto = (ventas["monto"]

.str.replace("$", "", regex=False)

.str.replace(",", "", regex=False)

.str.strip())

ventas["monto"] = pd.to_numeric(texto, errors="coerce")

anota("monto", antes, len(ventas), "montos a número")

# Only the sales that have an amount, to add up the total collected.

antes = len(ventas)

cobradas = ventas.dropna()

anota("con monto", antes, len(cobradas), "quita las ventas sin monto")

Run your code to see what it prints and the value of its last expression. Every run starts fresh. Ctrl+Enter runs it. Tab indents; to leave the editor, press Escape, then Tab.

Ready to run your code.

Run your code to see its output here.

Data: ventas-cooperativa, 60 rows

Enter saves a cell and Escape leaves it as it was. Leave a cell empty for a missing value.

The exercise's data, "ventas-cooperativa". Your edits run with your code.
Row										Remove
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

Page 1 of 3

Run it yourself

The original code leaves 14 of 60 sales, with no error. The cause is mostly descuento: 44 sales have it empty (which means zero, as Marisol confirmed), and dropna() without subset removed every sale with any blank: those, the two with no amount and the one with no kilos, 46 in all. With subset=["monto"] the step does what its note says.

Without the log, the total collected would have come out with fewer than a quarter of the sales, and the number would have looked perfectly normal.

Check yourself: In your log, the "normalizar ciudad" (normalize city) step says 60 -> 57. What do you conclude?

pandas: DataFrame.dropna (how and subset)