Eigma
EN 中 DE
What Is Inside a Chatbot Open Eigma
Contents Text as tokens

What the thing is

Text as tokens

A chat model does not read letters or words. Its input is a sequence of numbers, and each number stands for a chunk of text called a token.1 OpenAI's documentation describes these chunks as commonly occurring sequences of characters, which means a token is sometimes a whole short word and sometimes only a fragment of a longer one.1

The boundaries are not where a reader would put them. The documented example is the word tokenization, which comes apart into token and ization.1 OpenAI's own tokenizer library gives a second: encoding often becomes encod and ing.2 Hugging Face's reference, which covers the same technique in Llama, Gemma and Qwen as well as in OpenAI's models, gives a third: annoyingly may split as annoying and ly, or as annoy, ing and ly, depending on which tokenizer does the splitting.3

The list of tokens is built rather than designed. The algorithm starts from individual characters and repeatedly merges the most frequent adjacent pair until it reaches a target size, so the list is a statistical residue of whatever text it was built from.3 The idea is older than any chatbot: it was introduced in 2015 as a way of "encoding rare and unknown words as sequences of subword units".4 Neither obvious alternative survives contact with real text. A token per word makes the list enormous and still leaves the model helpless in front of a word it has never met. A token per character keeps the list small but makes every sequence far longer, and a single letter carries far less meaning than the word it sits in.3 Starting instead from the 256 possible byte values fixes the first problem, because then any text at all can be encoded, including scripts and emoji absent from the original corpus.3

In practice this is why a limit stated in tokens is not a word limit. OpenAI's rule of thumb puts one token at roughly four characters of English.1

The boundary. A token count belongs to the tokenizer, not to the text. Anthropic documents that its newer models use a newer tokenizer on which the same input produces about thirty percent more tokens than before, and that a counted total is an estimate the real request may differ from slightly.5 Sizes here are illustrative only: GPT-2's list held 50,257 entries,3 and vendors do not publish the size for a current model. The figure below lines up three of the documented splits from above, side by side, without claiming a token count for any whole sentence.

<svg viewBox='0 0 700 220' width='700' height='220' xmlns='http://www.w3.org/2000/svg' font-family='system-ui, sans-serif' fill='currentColor'>
  <title>Three documented examples of one whole word splitting into token fragments</title>
  <text x='12' y='24' font-size='14'>Documented splits, not a token count for any full sentence. Illustrative.</text>
  <text x='12' y='64' font-size='15'>tokenization</text>
  <rect x='190' y='46' width='88' height='34' rx='6' fill='none' stroke='currentColor' stroke-opacity='0.55'/>
  <text x='234' y='69' font-size='14' text-anchor='middle'>token</text>
  <rect x='286' y='46' width='96' height='34' rx='6' fill='none' stroke='currentColor' stroke-opacity='0.55'/>
  <text x='334' y='69' font-size='14' text-anchor='middle'>ization</text>
  <text x='12' y='114' font-size='15'>encoding</text>
  <rect x='190' y='96' width='88' height='34' rx='6' fill='none' stroke='currentColor' stroke-opacity='0.55'/>
  <text x='234' y='119' font-size='14' text-anchor='middle'>encod</text>
  <rect x='286' y='96' width='60' height='34' rx='6' fill='none' stroke='currentColor' stroke-opacity='0.55'/>
  <text x='316' y='119' font-size='14' text-anchor='middle'>ing</text>
  <text x='12' y='164' font-size='15'>annoyingly</text>
  <rect x='190' y='146' width='118' height='34' rx='6' fill='none' stroke='currentColor' stroke-opacity='0.55'/>
  <text x='249' y='169' font-size='14' text-anchor='middle'>annoying</text>
  <rect x='316' y='146' width='48' height='34' rx='6' fill='none' stroke='currentColor' stroke-opacity='0.55'/>
  <text x='340' y='169' font-size='14' text-anchor='middle'>ly</text>
  <text x='12' y='202' font-size='13'>Each box becomes one number. Where the split falls depends on the tokenizer, not the word.</text>
</svg>

Figure: Three documented word-to-token splits; no source gives a token count for a whole sentence.

References

  1. [1]OpenAI API concepts: tokens
  2. [2]tiktoken, OpenAI's tokenizer library
  3. [3]Hugging Face Transformers: tokenizer summary
  4. [4]Sennrich, Haddow and Birch, subword units, 2015
  5. [5]Anthropic: token counting
Quizzes

  1. What does the documented behaviour of tokenizers show about tokens and words?

    • The same text can split into different tokens depending on which tokenizer reads it
    • Every tokenizer aligns its token boundaries exactly with word boundaries
    • A word's split into tokens is fixed once and holds for any tokenizer

    Hugging Face's reference documents annoyingly splitting one way or another depending on the vocabulary, which shows the boundary belongs to the tokenizer rather than the word.

  2. A token count is a property of the ____ that reads the text, not of the text on its own.

    • model's tokenizer
    • reader's screen
    • writer's vocabulary

    Anthropic documents its newer tokenizer producing about thirty percent more tokens for the same input, which is only possible if the count belongs to the tokenizer.

  3. The same sentence can become a different quantity of tokens depending on which model reads it.

    • True
    • False

    True. The split is decided by the particular tokenizer, so one vendor's newer models turn identical text into roughly thirty percent more tokens than its older ones.

Comments

No comments yet. Start the conversation.