Skip to content
Clean Your Data Without Breaking It
Start free

Why doesn't drop_duplicates() remove my repeats? Define the key first

Predict what drop_duplicates() removes with different columns, tell three kinds of repeats apart, and choose the key that says when two rows are the same thing.

Free course. Your Maestro, an AI, .

The inventory found 2 extra folios in the sales sheet. The natural reaction is to run drop_duplicates() and move on. Before you do that on the full sheet, see what it does on a six-row table. It mixes three different situations, the way real sheets do:

  • Sale V-1 was entered twice, identically (entries 1 and 5).
  • Hotel Mirador placed two identical orders on the same day, with different folios (V-2 and V-3). That's two sales.
  • Sale V-4 was entered with 5 kilos and later corrected to 50 (entries 4 and 6).

The captura (entry) column is the row number each sale was entered on: it goes up as someone types into the sheet. Predict each result before you run it.

Without saying which columns

Example 1 of 4

example.py
import pandas as pd

ventas = pd.DataFrame({
    "captura": [1, 2, 3, 4, 5, 6],
    "folio": ["V-1", "V-2", "V-3", "V-4", "V-1", "V-4"],
    "cliente": ["Café La Esquina", "Hotel Mirador", "Hotel Mirador",
                "Grano Fino", "Café La Esquina", "Grano Fino"],
    "fecha": ["2026-03-02", "2026-03-02", "2026-03-02",
              "2026-03-05", "2026-03-02", "2026-03-05"],
    "kilos": [40, 163, 163, 5, 40, 50],
})

print(len(ventas.drop_duplicates()))

A guess is fine: it's kept as you wrote it, so you can compare. Only you and your Maestro see your predictions.

Example 2 of 4, opens once you explain this one

Make your prediction

Three things that look repeated

What the table showed you is that "duplicate" isn't one thing. There are three cases, and each calls for something different:

  1. The same thing entered twice, identically. V-1. One copy is extra, and it doesn't matter which.
  2. The same thing entered twice, differently. V-4: an entry and its correction. One copy is extra, and it does matter which (the next page).
  3. Two different things that look alike. V-2 and V-3. Nothing is extra. Removing one deletes a real sale.

No command tells these cases apart for you. What tells them apart is the key: the column or columns that say when two rows are the same thing. In the sales sheet, the key is the folio, because every sale has one and the cooperative never reuses it. Two rows with the same folio are the same sale; two with different folios are two sales, even if they match on everything else.

Choosing the key means asking yourself what a row represents, which is called the table's . In the sales sheet, a row is a sale. In the member list, a row is a member. If you can't say in one sentence what a row represents, you're not ready to remove duplicates.

Not sure what the key would be for a table at your job? Tell your Maestro what each row represents.

In the full March sheet, the same commands give: drop_duplicates() leaves all 60 sales; without the entry column, 59 (the second exact copy of V-0017 goes); with the folio, 58. And with customer, date and kilos you also get 58, but by deleting V-0041, Hotel Mirador's real second order, and leaving V-0031 twice. Same number, wrong table. That's why seeing how many rows are left isn't enough: you have to see which ones.

Try it

The member list has no folio. There's a socio_id, but one member was registered twice with two different ids, so the id won't help you find her. Look for repeats by name and then by name and municipality. If you don't see why the two results are different, ask your Maestro.

First normalize municipio (you already have the map). Then print the members who repeat by nombre and those who repeat by nombre and municipio, with duplicated(..., keep=False) to see every copy. You're done when you can say which of the two groups is the same person and which is

First normalize municipio (you already have the map). Then print the members who repeat by nombre and those who repeat by nombre and municipio, with duplicated(..., keep=False) to see every copy. You're done when you can say which of the two groups is the same person and which is two different people.

Run your code to see what it prints and the value of its last expression. Every run starts fresh.

Ready to run your code.

Run your code to see its output here.

Data: socios-cooperativa, 30 rows

Enter saves a cell and Escape leaves it as it was. Leave a cell empty for a missing value.

The exercise's data, "socios-cooperativa". Your edits run with your code.
RowRemove
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Page 1 of 2

Run it yourself

By name, 4 rows come up: Juana Martínez Cruz twice and José Luis Pérez Santiago twice. By name and municipality, only Juana comes up. The José Luis Pérez Santiago from San Agustín Loxicha and the one from Candelaria Loxicha are two different growers, with different plots and phone numbers: two people can have the same name. With the name as the key, you would have deleted one of them from the member list.

And notice the order: if you don't normalize the municipality first, Juana doesn't come up either, because one copy says "Pluma Hidalgo" and the other "pluma hidalgo". Cleaning the text comes before looking for duplicates. Even so, Juana showing up is a suspicion, not proof: Marisol confirms it, and that's the next page.

Check yourself: In the sales sheet, someone removes duplicates with `subset=["cliente", "fecha", "kilos"]` and is left with 58 sales, the same as with the folio. Is that fine?

pandas: DataFrame.drop_duplicates (subset and keep) (opens in a new tab)