seif.dev
BITE 2 · 4 MIN · +10 XP

What are tokens in AI?

Updated Topics:How AI models work

Tokens are the pieces a model reads and writes instead of words. Before the model sees your text, a tokenizer cuts it into tokens, and each token has an id: a number from a fixed list. Common words are often one token; rare or long words are split into smaller pieces. Models count, bill and remember in tokens.

Try it. Pick a word and see how it splits.

TOKENIZER · GPT-4O

Pick a word. See its tokens.

  • un373
  • bel9880
  • ievable45794

The ids are what the model works with. It reads a row of numbers and guesses the id of the next token. The tokenizer turns the ids back into text for you.

These splits come from the tokenizer GPT-4o uses (o200k_base). Other models use other lists, so the same word can split differently.

QUIZ

Is "unbelievable" 3 tokens because it has 3 syllables?

Tokens also explain limits you will meet: models count, bill and remember in tokens, not words.

Keep your XP.

Save your progress and hear when new puzzles and lessons are added.