It reads tokens, not words

A model never sees letters or words. Before it reads your text, a tokenizer cuts it into pieces called tokens, and each token has an id: a number from a fixed list.

Common words are often one token. Rare or long words are cut into smaller pieces the list does have.

Turn on JavaScript to play this bite.

The ids are what the model works with. It reads a row of numbers and guesses the id of the next token. The tokenizer turns the ids back into text for you.

These splits come from the tokenizer GPT-4o uses (o200k_base). Other models use other lists, so the same word can split differently.

Tokens also explain limits you will meet: models count, bill and remember in tokens, not words.