Tokens are the pieces a model reads and writes instead of words. Before the model sees your text, a tokenizer cuts it into tokens, and each token has an id: a number from a fixed list. Common words are often one token; rare or long words are split into smaller pieces. Models count, bill and remember in tokens.
Try it. Pick a word and see how it splits.
Turn on JavaScript to play this bite.
Pick a word. See its tokens.
- un373
- bel9880
- ievable45794
The ids are what the model works with. It reads a row of numbers and guesses the id of the next token. The tokenizer turns the ids back into text for you.
These splits come from the tokenizer GPT-4o uses (o200k_base). Other models use other lists, so the same word can split differently.
Is "unbelievable" 3 tokens because it has 3 syllables?
Tokens also explain limits you will meet: models count, bill and remember in tokens, not words.