What is UTF-8, and why does an "é" take two bytes?

To understand "Mérida" you have to go one level down, to what's really on the disk. A computer doesn't store letters: it stores bytes, numbers from 0 to 255. To store text it needs a table that says which byte or bytes stand for each letter. That table is called an encoding.

Two common encodings in these examples are UTF-8, which can write the characters of any language, and Windows-1252, an encoding that uses one byte per representable character used by many older programs and systems on Windows. In Python they're called "utf-8" and "cp1252". For ASCII characters, such as the unaccented Latin letters in these examples, the two store the same bytes. For é, ñ and ü, they don't.

Before any explanation, predict. In each program, len counts how many items something has, and .encode(...) turns text into its bytes using the table you name. Write down what you think each one prints and why; then run it.

Here's what you saw, in one table, with what Python printed:

Three ideas to hold on to:

1. The encoding belongs to the file, not to the letter. The é is the same letter in both cases; what changes is how it was written as bytes. A file doesn't store "accented letters", it stores the bytes a table chose.
2. Without accents, you can't tell. A file with only names like Juan or Tonala looks the same in both tables. That's why the problem shows up out of nowhere, the day the first Tonalá or Peña arrives.
3. Reading means applying a table. To turn bytes into letters, the program has to use the same table the file was written with. The file doesn't tell it which. If it uses another one, other letters come out: that's the next page.

Latin-1 (also called ISO-8859-1) is almost a twin of Windows-1252: for Spanish letters they give the same bytes, and they differ in symbols like the euro. If a program offers "Latin-1" for reading a Windows-1252 file with Spanish text, it usually works. If you can't keep them straight, ask your Maestro.

Try it

Replace the word in the program with made-up names, such as Ciudad del Peñón or Papelería La Cigüeña. Don't use real customer names or confidential information. Find one that takes the same bytes in both tables and one that doesn't. Try an emoji too, and read what Python tells you.

When the letter doesn't fit in the table

So far both tables knew every letter you tried. But Windows-1252 gives each letter a single byte, and one byte can only tell 256 values apart: there's no room for everything that exists. UTF-8 has room, because it uses more bytes when it needs to. Try a symbol that isn't a Spanish letter, a coffee cup next to the word:

UTF-8 wrote the cup as three bytes, E2 98 95, and Windows-1252 couldn't: Python stops with an error instead of writing something else in its place. That's why, when you write a file, UTF-8 is the safe bet, and Windows-1252 only makes sense if the system that will read it requires it. You'll come back to this in module 4. If your file has symbols like the euro or an emoji, ask your Maestro which encoding to use.