How does "Mérida" turn into "Mérida"?
Follow a word's bytes from one program to another, recognize the two kinds of broken accents and repair a column that arrived that way.
Free course. Your Maestro, an AI, .
You already know that "Mérida" saved in UTF-8 is seven bytes: 4D C3 A9 72 69 64 61. Five are unaccented letters and two, C3 A9, are the é. Now follow those bytes when another program opens the file.
Someone types Mérida
in the order system
Saved as UTF-8
4D C3 A9 72 69 64 61Another program opens the file
Path 1 of 2: reads it as UTF-8
You see Mérida
C3 A9are read together: é
Ends here.
Path 2 of 2: reads it as Windows-1252
You see Mérida
C3is read as à andA9is read as ©
Ends here.
Windows-1252 gives one byte to each letter, so it reads C3 as one letter (Ã) and A9 as another (©). The é turns into two characters, and the first one is almost always an Ã. This has a name: mojibake, the text you get when bytes are read with the wrong table. This is how some of El Nopal's customers' cities look when a UTF-8 file is read as Windows-1252 (we ran it in Python):
| Written | Read |
|---|---|
| Querétaro | Querétaro |
| León | León |
| Mazatlán | Mazatlán |
| Tlajomulco de Zúñiga | Tlajomulco de Zúñiga |
| Miscelánea El Güero | Miscelánea El Güero |
The pattern is so regular you can spot it from far away: a sequence like é is consistent with UTF-8 read as Windows-1252 or Latin-1. It is a clue to check, not a unique identification of the encoding.
The road backwards
Now the opposite case: a branch's old system saves in Windows-1252, and a program that expects UTF-8 opens it. The é is a single byte, E9, and in UTF-8 that lone byte doesn't form any valid letter. The program has two ways out: stop with an error, or put the replacement sign, �, in its place and keep going. That's how you get "M�rida".
That diamond with a question mark is worse news than the Ã. The program already threw away the byte it didn't understand, so from that text there's no way to know which letter it was. The fix is to go back to the original file and read it with the right encoding. pandas, for example, takes the first way out: it stops. Predict what happens:
pandas opens a file from the old system
Example 1 of 2
import pandas as pd# This is how the branch's old system saves the file: in Windows-1252.with open("clientes.csv", "w", encoding="cp1252") as archivo: archivo.write("cliente,ciudad\nPapelería La Única,Mérida\nMiscelánea El Güero,León\n")print(pd.read_csv("clientes.csv"))A guess is fine. What you write stays in this browser and waits in your course.
Example 2 of 2, opens once you explain this one
When the text already arrived broken
Sometimes you don't have the original file. Accounting opened the CSV in Excel, saw "Mérida", and saved it like that; now the broken text is the content. If you confirm that chain of conversions and no bytes were lost, you can try to undo it. An à on its own doesn't guarantee that this recipe is the right one. Follow the reasoning step by step; try each step before opening it.
Repairing a name that arrived as Miscelánea El Güero
Worked example · 4 steps
In the customer column of the accounting report you see "Miscelánea El Güero". You don't have the original file. How do you recover the name without retyping it by hand?
Work out the next step yourself, then open it to compare.
Try it
Now the whole column. Change a single line so every name comes out repaired. If you don't know which line to change, ask your Maestro for a hint.
The column in this exercise contains only examples with the same known conversion. In a mixed column, separate and review the cases before applying the repair; always keep the original.
Your turn
These are the customers from the accounting report, exactly as they arrived. Change the reparado line so every name comes out spelled right. You're done when it prints Papelería La Única, Miscelánea El Güero, Abarrotes Doña Peña and Farmacia San José.
Tab indents; Esc then Tab leaves
Common mistake: Repairing is the last resort
If you have the original file, don't repair text: read it with the right encoding (encoding="utf-8" or encoding="cp1252"). Repairing is for what someone already saved broken, and only once per text.
A quick guide for deciding
The two symptoms on this page are easy to spot, and each calls for something different:
| What you see | What might have happened | What you check |
|---|---|---|
à next to another letter (é, ó) | UTF-8 may have been read as Windows-1252 or Latin-1 | Read it again with encoding="utf-8"; if it's already saved that way, repair it with encode("cp1252").decode("utf-8") |
� where the letter should be | A conversion replaced bytes it couldn't interpret; the symbol doesn't identify the original encoding | Go back to the original file and confirm its encoding before reading it |
UnicodeDecodeError when reading | A program that expects UTF-8 ran into bytes that aren't | Check with the source; if it specifies Windows-1252, try encoding="cp1252" and verify several names and symbols |
An honest warning: repair isn't always possible. Í in UTF-8 is the bytes C3 8D, and in the sandbox "Í".encode("utf-8").decode("cp1252") fails, because Windows-1252 leaves the byte 8D undefined. If you run into a name like that, the way out is the usual one: go back to the original file. If you can't tell which of the three cases you're in, paste what you see to your Maestro.
Check yourself: A branch report shows "Le�n", with a diamond and a question mark. What happened, and what do you do?