How does "Mérida" turn into "Mérida"?
Follow a word's bytes from one program to another, recognize the two kinds of broken accents and repair a column that arrived that way.
Free course. Your Maestro, an AI, .
You already know that "Mérida" saved in UTF-8 is seven bytes: 4D C3 A9 72 69 64 61. Five are unaccented letters and two, C3 A9, are the é. Now follow those bytes when another program opens the file.
Someone types Mérida
in the order system
Saved as UTF-8
4D C3 A9 72 69 64 61Another program opens the file
Path 1 of 2: reads it as UTF-8
You see Mérida
C3 A9are read together: é
Ends here.
Path 2 of 2: reads it as Windows-1252
You see Mérida
C3is read as à andA9is read as ©
Ends here.
Windows-1252 gives one byte to each letter, so it reads C3 as one letter (Ã) and A9 as another (©). The é turns into two characters, and the first one is almost always an Ã. This has a name: mojibake, the text you get when bytes are read with the wrong table. This is how some of El Nopal's customers' cities look when a UTF-8 file is read as Windows-1252 (we ran it in Python):
| Written | Read |
|---|---|
| Querétaro | Querétaro |
| León | León |
| Mazatlán | Mazatlán |
| Tlajomulco de Zúñiga | Tlajomulco de Zúñiga |
| Miscelánea El Güero | Miscelánea El Güero |
The pattern is so regular you can spot it from far away: an à where an accented letter should be means "this was UTF-8 and someone read it as Windows-1252".
The road backwards
Now the opposite case: a branch's old system saves in Windows-1252, and a program that expects UTF-8 opens it. The é is a single byte, E9, and in UTF-8 that lone byte doesn't form any valid letter. The program has two ways out: stop with an error, or put the replacement sign, �, in its place and keep going. That's how you get "M�rida".
That diamond with a question mark is worse news than the Ã. The program already threw away the byte it didn't understand, so from that text there's no way to know which letter it was. The fix is to go back to the original file and read it with the right encoding. pandas, for example, takes the first way out: it stops. Predict what happens:
pandas opens a file from the old system
Example 1 of 2
import pandas as pd
# This is how the branch's old system saves the file: in Windows-1252.
with open("clientes.csv", "w", encoding="cp1252") as archivo:
archivo.write("cliente,ciudad\nPapelería La Única,Mérida\nMiscelánea El Güero,León\n")
print(pd.read_csv("clientes.csv"))
A guess is fine: it's kept as you wrote it, so you can compare. Only you and your Maestro see your predictions.
Example 2 of 2, opens once you explain this one
When the text already arrived broken
Sometimes you don't have the original file. Accounting opened the CSV in Excel, saw "Mérida", and saved it like that; now the broken text is the content. If the symptom is the Ã, it can be repaired, because the wrong reading didn't throw away any byte: it only displayed them as other letters. Follow the reasoning step by step; try each step before opening it.
Repairing a name that arrived as Miscelánea El Güero
Worked example · 4 steps
In the customer column of the accounting report you see "Miscelánea El Güero". You don't have the original file. How do you recover the name without retyping it by hand?
Work out the next step yourself, then open it to compare.
Try it
Now the whole column. Change a single line so every name comes out repaired. If you don't know which line to change, ask your Maestro for a hint.
These are the customers from the accounting report, exactly as they arrived. Change the reparado line so every name comes out spelled right. You're done when it prints Papelería La Única, Miscelánea El Güero, Abarrotes Doña Peña and Farmacia San José.
These are the customers from the accounting report, exactly as they arrived. Change the reparado line so every name comes out spelled right. You're done when it prints Papelería La Única, Miscelánea El Güero, Abarrotes Doña Peña and Farmacia San José.
Run your code to see what it prints and the value of its last expression. Every run starts fresh. Ctrl+Enter runs it. Tab indents; to leave the editor, press Escape, then Tab.
Ready to run your code.
Run your code to see its output here.
Common mistake: Repairing is the last resort
If you have the original file, don't repair text: read it with the right encoding (encoding="utf-8" or encoding="cp1252"). Repairing is for what someone already saved broken, and only once per text.
A quick guide for deciding
The two symptoms on this page are easy to spot, and each calls for something different:
| What you see | What happened | What you do |
|---|---|---|
à next to another letter (é, ó) | A UTF-8 file was read as Windows-1252 | Read it again with encoding="utf-8"; if it's already saved that way, repair it with encode("cp1252").decode("utf-8") |
� where the letter should be | A Windows-1252 file was read as UTF-8 and the program threw away the byte it didn't understand | Go back to the file and read it with encoding="cp1252" |
UnicodeDecodeError when reading | A program that expects UTF-8 ran into bytes that aren't | Try encoding="cp1252" and check that the names read correctly |
An honest warning: repair isn't always possible. Í in UTF-8 is the bytes C3 8D, and in the sandbox "Í".encode("utf-8").decode("cp1252") fails, because Windows-1252 leaves the byte 8D undefined. If you run into a name like that, the way out is the usual one: go back to the original file. If you can't tell which of the three cases you're in, paste what you see to your Maestro.