What is a BOM, and why does my first column have extra characters?
Meet the three invisible bytes at the start of some UTF-8 files: they help Excel and get in your script's way. Learn to read past them.
Free course. Your Maestro, an AI, .
In the course's first practice, the one with ver.py, you saw the price list start oddly: its first field wasn't plain sku. Python showed an escape in front of it, a backslash followed by ufeff, which is how it writes a character you can't see. That extra, invisible character is the BOM (short for byte order mark). In a UTF-8 file it's three bytes at the very start, EF BB BF, before the first letter. It isn't data: it's a mark that says "this is UTF-8."
Remember that a file doesn't announce its encoding. The BOM is the closest thing to an announcement, which is why Excel cares about it. Microsoft says so in its help: "You can open a CSV file encoded with UTF-8 normally if it was saved with BOM (Byte Order Mark)"; if not, you have to import it and tell it the encoding. And for Power Query, Excel's data import tool, it documents that the character set isn't inferred and "UTF-8 is only inferred if it starts with a UTF-8 BOM."
So the BOM has two faces. For Excel it's a help: without it, your perfectly good UTF-8 file can open with à in the accents when you double-click it. For other programs it's an extra character that sneaks into the name of the first column. Predict:
Each program's file is created with a BOM at the start (utf-8-sig writes it that way). Then it's read in different ways.
Each program's file is created with a BOM at the start (utf-8-sig writes it that way). Then it's read in different ways.
Python's csv module, with utf-8
Example 1 of 4
import csv
with open("precios.csv", "w", encoding="utf-8-sig") as archivo:
archivo.write("sku,precio\n00731,23.50\n")
with open("precios.csv", encoding="utf-8") as archivo:
fila = next(csv.DictReader(archivo))
print(fila)
print(fila["sku"])
A guess is fine: it's kept as you wrote it, so you can compare. Only you and your Maestro see your predictions.
Example 2 of 4, opens once you explain this one
A character you can't see, but that counts
That's why the BOM is so baffling: with a BOM and without one, the name sku looks the same on screen, and yet Python treats them as different names. There are two ways to catch it that don't depend on your eyes: read the file's first bytes in binary mode ("rb") and ask Python for the name's repr, which writes out invisible characters. This is what a program run in the sandbox printed:
with open("precios.csv", "w", encoding="utf-8-sig") as archivo:
archivo.write("sku,precio\n00731,23.50\n")
with open("precios.csv", "rb") as archivo:
print(archivo.read(3))
with open("precios.csv", encoding="utf-8") as archivo:
primera = archivo.readline().split(",")[0]
print(len(primera), "caracteres")
print(repr(primera))
b'\xef\xbb\xbf'4 caracteres'\ufeffsku'What looks like sku has 4 characters, not 3, and repr gives it away: the last line shows the escape for the invisible character in front of sku. If a KeyError says a column doesn't exist and you can see it right there, print repr(name) before looking for another cause. Ask your Maestro to help you check whether one of your files starts with a BOM.
The practical rule in Python: when you read a CSV that might come from Excel or another system, use encoding="utf-8-sig". It reads a file with a BOM and one without it the same way, so you lose nothing. In pandas you don't need it for the BOM, but writing it makes clear what you expect.
And when the problem is the other way around, when you write a file for someone who will double-click it open in Excel, the BOM works in your favor: you'll see that in module 4, on the export page.
Do you have a script that fails with KeyError on the first column? Tell your Maestro. The first thing it will ask you is which encoding you open the file with.
Try it
This script adds up the inventory from the price list with the csv module. Run it: it fails. Fix it by changing just one thing.
Run the program and read the error. Change only the encoding the file is read with (not the line that creates it, which imitates how it arrives). You're done when it prints the units in stock for each product and the total: 1644 units.
Run the program and read the error. Change only the encoding the file is read with (not the line that creates it, which imitates how it arrives). You're done when it prints the units in stock for each product and the total: 1644 units.
Run your code to see what it prints and the value of its last expression. Every run starts fresh. Ctrl+Enter runs it. Tab indents; to leave the editor, press Escape, then Tab.
Ready to run your code.
Run your code to see its output here.
Tip: Three ways to see the same BOM
As bytes: EF BB BF, or b'\xef\xbb\xbf' in Python. Read as UTF-8 without removing it: one invisible character, U+FEFF, which Python's repr writes as a backslash and ufeff, stuck to the first name. Read as Windows-1252: three odd letters, with an ï at the start and a ¿ at the end. All three are the same mark.