Tokenization is the preliminary step to text processing in a language model: it consists of dividing a sequence of characters into discrete units called tokens, which the model then converts into numerical values. A token does not necessarily equal a word; it can be a complete word, a word fragment (subword), a punctuation mark, or even a single character, depending on the algorithm used.
It matters because it influences both performance and cost. Models have a context window limit measured in tokens, and many services bill by this unit. Furthermore, a good tokenization strategy allows for handling unknown words without discarding them. The most common methods are:
- BPE (Byte Pair Encoding), used in the GPT family.
- WordPiece, employed by BERT.
- SentencePiece, frequent in multilingual models.
A practical nuance: in Spanish, long or uncommon words may be fragmented into several tokens, so the same text usually consumes more tokens than its English equivalent, affecting the cost and the available space in the context.