In short
LLMs don't read letters or words. They read tokens, common chunks of text that each have an ID number. Tokens decide how much a model can take in at once, its context window, and how much each answer costs.
Type "I am learning how AI works." into the tokenizer that GPT-4o uses, and it cuts the sentence into 7 pieces. Type the same sentence in Hindi, "मैं सीख रहा हूँ कि AI कैसे काम करता है।", and you get 23 pieces. Same meaning, more than three times as many pieces.
Those pieces are tokens, and they matter more than you'd think. They decide how much an AI can read at once, how quickly it answers and how much it costs. In this lesson you'll see what tokens are, how text turns into numbers, why some languages need more tokens than others, and what "context window" and "per token" pricing mean.
What is a token?
A token is a chunk of text that a model treats as one unit. It can be a whole word, part of a word, a punctuation mark, or a word with the space before it. Splitting text into tokens is called tokenization, and the program that does it is a tokenizer. As you saw in lesson 4, the full list of tokens a model knows is its vocabulary. GPT-4o's tokenizer has about 200,000.
Here's how GPT-4o's tokenizer splits a few words:
- " unbelievable" (with a space in front) is a single token, because it's common in English text.
- "Bengaluru" becomes four tokens: "B", "eng", "al" and "uru".
- "strawberry" becomes three: "st", "raw" and "berry".
As a rule of thumb, OpenAI says one token is about four characters of English, or three-quarters of a word. So 100 tokens is roughly 75 words.
Why not just use words, or letters?
Whole words would need a vocabulary with every word, name and typo ever written, and a brand-new word would have no token at all. Single letters would need only a tiny vocabulary, but every sentence would become very long, which makes the model slower and more expensive to run. Tokens sit in between: common words get a token of their own, and rare words are built from pieces, so any text can be written with a vocabulary of a few hundred thousand tokens.
Most tokenizers today learn their vocabulary with byte-pair encoding (BPE): start with single characters, then again and again merge the pair that appears together most often in a huge pile of text, until the vocabulary is big enough. That's why common words end up as one token and rare ones stay in pieces.
From tokens to numbers
The model never sees text at all. Every token in the vocabulary has an ID, a number, and the tokenizer turns your text into a list of these numbers before the model reads it. When the model answers, it picks token IDs (that's the loop from lesson 4), and the tokenizer turns them back into text for you.
I am learning how AI works.
- I, ID 40
- space then am, ID 939
- space then learning, ID 7524
- space then how, ID 1495
- space then AI, ID 20837
- space then works, ID 5882
- ., ID 13
Notice that most tokens include the space before the word. Small differences matter too: to GPT-4o's tokenizer, "Hello" is token 13225, " hello" with a space in front is token 40617, and "HELLO" is two tokens, "HEL" and "LO". For the model, these are different things.
Why Hindi needs more tokens
A tokenizer learns its vocabulary from the text it's trained on, and for most tokenizers, much of that text is English. So English words tend to get tokens of their own, while languages in other scripts, such as Hindi in Devanagari, are more often split into small pieces, sometimes a single letter or vowel sign at a time.
मैं सीख रहा हूँ कि AI कैसे काम करता है।
- म
- ैं
- space then स
- ी
- ख
- space then रह
- ा
- space then ह
- ूँ
- space then क
- ि
- space then AI
- space then क
- ै
- स
- े
- space then क
- ाम
- space then करत
- ा
- space then ह
- ै
- ।
Here's how many tokens the two sentences take with three generations of OpenAI tokenizers:
| Tokenizer | English | Hindi |
|---|---|---|
| GPT-2 (2019) | 7 | 54 |
| GPT-4 (2023) | 7 | 37 |
| GPT-4o (2024) | 7 | 23 |
It's getting better, but the gap is still there. A 2023 study found that the same text can need up to 15 times as many tokens in some languages as in others. When OpenAI introduced GPT-4o's tokenizer, it said a Hindi sample needed 2.9 times fewer tokens than before (31 instead of 90), with similar savings for Marathi, Tamil, Telugu and Gujarati.
This matters because more tokens means a higher bill, slower answers and less room for your text, which brings us to the context window.
The context window
A model's context window is the most tokens it can work with at once: your question, the conversation so far and its own answer, all together. Context windows have grown fast:
- GPT-2 (2019): 1,024 tokens, about two and a half pages.
- Llama 3.1 (2024): about 128,000 tokens, roughly a 300-page book.
- Gemini 1.5 Pro (2024): up to 1 million tokens, which Google said is over 700,000 words.
Here's something most people don't know: the model doesn't remember your earlier messages by itself. Behind the scenes, a chat app sends the whole conversation so far along with each new message. So a long chat keeps growing, and once it's bigger than the context window, the app has to drop or summarise the oldest parts. That's why a chatbot can "forget" something you told it at the start of a very long conversation.
Tokens and money
AI companies charge developers by the token, usually quoted per million tokens. There's one price for input tokens (what you send) and a higher one for output tokens (what the model writes). Anthropic's price list, for example, charges five times as much per output token as per input token.
So everything that adds tokens adds cost: a long prompt, a long chat that's sent again with every message, an answer that rambles, or a language that needs more tokens for the same meaning. That's why developers count tokens when they plan an app.
Odd behaviour that tokens explain
Tokens explain some well-known chatbot slips. Ask a model how many r's are in "strawberry", and it has often got it wrong. It doesn't see ten letters. It sees "st", "raw" and "berry", three IDs, and has to guess what letters are inside them. The same goes for spelling words backwards or counting letters in general: these are easy for you because you see letters, and hard for a model because it sees tokens.
Key takeaways
- LLMs read and write tokens: chunks of text such as words, parts of words or punctuation.
- Every token has an ID number, and the model only ever works with those numbers.
- In English, a token is about three-quarters of a word. Many other languages, including Hindi, need more tokens for the same meaning.
- The context window is how many tokens a model can handle at once. Long chats eventually spill over it.
- AI companies charge per token, and output tokens cost more than input tokens.
Check your understanding
Sources and further reading
- What are tokens and how to count them?, docs (opens in a new tab)Docs · OpenAI's rule of thumb that a token is about four characters, or three-quarters of an English word.
- Byte-Pair Encoding tokenization, course (opens in a new tab)Course · Hugging Face's step-by-step walk through how a tokenizer learns its vocabulary by merging common pairs.
- Let's build the GPT Tokenizer, video (opens in a new tab)Video · Andrej Karpathy builds a GPT tokenizer from scratch in about two hours, and explains many of its odd effects.
- Language Model Tokenizers Introduce Unfairness Between Languages, paper (opens in a new tab)Paper · The 2023 study showing the same text can need up to 15 times as many tokens in some languages.
- Hello GPT-4o, article (opens in a new tab)Article · OpenAI's 2024 announcement, whose "Language tokenization" section shows big token savings for Hindi, Marathi, Tamil, Telugu and Gujarati.
- Pricing, docs (opens in a new tab)Docs · An example of an AI company's price list, charged per million input and output tokens.