What is UTF-8, and why does an "é" take two bytes?
Predict how many bytes an accented letter takes in UTF-8 and in Windows-1252, then run it and see the difference.
Free course. Your Maestro, an AI, .
To understand "Mérida" you have to go one level down, to what's really on the disk. A computer doesn't store letters: it stores bytes, numbers from 0 to 255. To store text it needs a table that says which byte or bytes stand for each letter. That table is called an encoding.
The two you'll run into at work are UTF-8, which can write the characters of any language, and Windows-1252, a one-byte-per-letter table used by many older programs and systems on Windows. In Python they're called "utf-8" and "cp1252". For letters without accents, the two tables store exactly the same bytes. For é, ñ and ü, they don't.
Before any explanation, predict. In each program, len counts how many items something has, and .encode(...) turns text into its bytes using the table you name. Write down what you think each one prints and why; then run it.
Four short programs. Each one changes just one thing from the one before.
Four short programs. Each one changes just one thing from the one before.
One letter in UTF-8
Example 1 of 4
letra = "é"
print(len(letra))
print(letra.encode("utf-8"))
print(len(letra.encode("utf-8")))
A guess is fine: it's kept as you wrote it, so you can compare. Only you and your Maestro see your predictions.
Example 2 of 4, opens once you explain this one
Here's what you saw, in one table, with what Python printed:
| Text | UTF-8 | Windows-1252 |
|---|---|---|
e | b'e', 1 byte | b'e', 1 byte |
é | b'\xc3\xa9', 2 bytes | b'\xe9', 1 byte |
ñ | b'\xc3\xb1', 2 bytes | b'\xf1', 1 byte |
Mérida | 7 bytes | 6 bytes |
Three ideas to hold on to:
- The encoding belongs to the file, not to the letter. The é is the same letter in both cases; what changes is how it was written as bytes. A file doesn't store "accented letters", it stores the bytes a table chose.
- Without accents, you can't tell. A file with only names like
JuanorTonalalooks the same in both tables. That's why the problem shows up out of nowhere, the day the firstTonaláorPeñaarrives. - Reading means applying a table. To turn bytes into letters, the program has to use the same table the file was written with. The file doesn't tell it which. If it uses another one, other letters come out: that's the next page.
Latin-1 (also called ISO-8859-1) is almost a twin of Windows-1252: for Spanish letters they give the same bytes, and they differ in symbols like the euro. If a program offers "Latin-1" for reading a Windows-1252 file with Spanish text, it usually works. If you can't keep them straight, ask your Maestro.
Try it
Replace the word in the program with names from your work: your city, your last name, a customer's name. Find one that takes the same bytes in both tables and one that doesn't. Try an emoji too, and read what Python tells you.
Run the program. Then change palabra to "Tlajomulco de Zúñiga" and run it again: how many more bytes does it take in UTF-8 than in Windows-1252, and why? Try a word without accents and an emoji. You're done when you can explain in your own words where each difference comes from.
Run the program. Then change palabra to "Tlajomulco de Zúñiga" and run it again: how many more bytes does it take in UTF-8 than in Windows-1252, and why? Try a word without accents and an emoji. You're done when you can explain in your own words where each difference comes from.
Run your code to see what it prints and the value of its last expression. Every run starts fresh. Ctrl+Enter runs it. Tab indents; to leave the editor, press Escape, then Tab.
Ready to run your code.
Run your code to see its output here.
When the letter doesn't fit in the table
So far both tables knew every letter you tried. But Windows-1252 gives each letter a single byte, and one byte can only tell 256 values apart: there's no room for everything that exists. UTF-8 has room, because it uses more bytes when it needs to. Try a symbol that isn't a Spanish letter, a coffee cup next to the word:
palabra = "Café ☕"
print(palabra, "tiene", len(palabra), "letras")
print("UTF-8: ", palabra.encode("utf-8"), len(palabra.encode("utf-8")), "bytes")
try:
print("Windows-1252:", palabra.encode("cp1252"))
except UnicodeEncodeError as error:
print("Windows-1252: no cabe,", error.reason)
Café ☕ tiene 6 letras
UTF-8: b'Caf\xc3\xa9 \xe2\x98\x95' 9 bytes
Windows-1252: no cabe, character maps to <undefined>
UTF-8 wrote the cup as three bytes, E2 98 95, and Windows-1252 couldn't: Python stops with an error instead of writing something else in its place. That's why, when you write a file, UTF-8 is the safe bet, and Windows-1252 only makes sense if the system that will read it requires it. You'll come back to this in module 4. If your file has symbols like the euro or an emoji, ask your Maestro which encoding to use.