Token (in AI)
A token is the piece of text a language model reads and writes in: sometimes a whole word, sometimes part of a word, a number or a punctuation mark. Models count their limits and their cost in tokens.
Select any words here to ask your Maestro about them.
A model reads the question "How many customers came back in October?" How many tokens do you think that is?
Why it's worth knowing: language models measure everything in tokens. What fits in their context window, what each request costs in tools that bill by usage, and how long an answer can run are all counted in tokens, not words.
A rough rule: in English, a token is about three quarters of a word. Other languages, Spanish among them, usually take more tokens per word, because tokenizers were trained on more English text. A 10-page document runs somewhere between 5,000 and 8,000 tokens.
One consequence matters with data: an LLM doesn't see 48,250.75 as a number but as five pieces of text. That's why it can slip when it adds or compares long figures from memory. When the math matters, ask it to do the math in code, or redo it yourself.
Heads-up: elsewhere, token also means an access key (like the one your bank uses) or a unit of a cryptocurrency. Here it means the pieces of text.
In Your AI and your data, you practice asking an AI for math you can check.
How it's used
How many tokens your CSV is
sales_march.csv
- date,item,quantity,total
- 2026-03-01,Screw 1/4 in,12,54.00One row: 20 tokens. Dates and numbers break into lots of pieces.
- marked: ... 4,811 rows: about 96,000 tokensIt fits in a model with a 200,000-token window, but fills almost half of it before you've asked anything.
When to look closer
Why does AI slip when counting letters?
How many r's are in "strawberry"?
- strawberry
If the model doesn't see letters, what does it see?
Ask your Maestro about Token
It answers from the free course page closest to this word, the way it answers while you study.
You: What does "Token" mean, in plain words?
Your Maestro: I'll explain it with a case of yours: it sticks better than a definition. Has an AI ever slipped on you while adding up long numbers?
Asking about: Answers from the page "How do you define a KPI? Write its definition sheet"
Only this browser keeps what you ask here. How it works
Learn more
Related terms
- LLM (large language model)
An LLM is a program trained on huge amounts of text to continue a piece of text one chunk at a time: it works out which chunk is likeliest to come next and writes it. ChatGPT, Claude and Gemini all run on an LLM.
- Context window
Everything a language model has in front of it at a given moment: your instruction, the files it has read and the results of its tools. What isn't in the context, the model can't use.
- Prompt
A prompt is what you write to an AI model to ask it for something: the question or instruction, plus everything you give it to work with, like data, examples or the format you want back.