Tokenization Strategies: A Comparative Analysis

The Spectrum of Tokenization

The core challenge of tokenization is to find the right level of granularity. We want units that are meaningful enough to learn from but flexible enough to represent any possible text. There are three main approaches, which exist on a spectrum from coarse to fine-grained.

A visual comparison of word-based, subword, and character-based tokenization. Each approach breaks down words like "Learning" and "Learned" into different fundamental units.

As we analyze these methods, we'll see that each represents a different point in a fundamental trade-off.

1. Word-Level Tokenization: The Intuitive Approach

The most straightforward way to tokenize text is to split it into words, typically using spaces and punctuation as delimiters. This is how humans process language, so it feels like a natural starting point.

Advantages:

• Semantic Units: Words are the primary carriers of meaning in language. Each token is a complete semantic unit, which is a strong starting point for the model.
• Shorter Sequences: A sentence is represented by a relatively small number of tokens, which keeps computational requirements manageable.

Disadvantages:
However, this simplicity comes with significant drawbacks, particularly when dealing with the vast and ever-changing nature of language.

• Vocabulary Size: The number of unique words in a language is enormous. A model trained on a large corpus like the web might encounter millions of unique words. This creates an impractically large vocabulary, leading to a massive embedding layer and an equally massive final output layer, demanding huge amounts of memory and computational power.
• Out-of-Vocabulary (OOV) Words: No matter how large your training vocabulary is, you will always encounter words at test time that you haven't seen before. These could be new slang, technical jargon, names, or simple misspellings. Word-level models handle this by mapping all unknown words to a single "unknown" token, often represented as <UNK>. This is a major information bottleneck; the model has no way of distinguishing between antidisestablishmentarianism and goooood, as both would become <UNK>.

The problem of OOV words is especially severe in morphologically rich languages, where a single root word can have hundreds of inflected forms.

2. Character-Level Tokenization: The Flexible Extreme

At the opposite end of the spectrum is character-level tokenization. Here, the text is simply treated as a sequence of characters.

Advantages:

• Small Vocabulary: The vocabulary is incredibly small and finite, consisting of all possible characters (letters, numbers, punctuation). For English, this is less than 100 tokens.
• No OOV Problem: Since any word is just a sequence of characters, the concept of an "unknown" word disappears. The model can, in principle, represent any text, including misspellings, neologisms, and complex morphological forms.

Disadvantages:
This flexibility, however, introduces its own set of severe challenges.

• Extremely Long Sequences: A sentence that is 20 words long might become a sequence of over 100 characters. As the computational cost of models like the Transformer scales quadratically with sequence length, this makes character-level models extremely slow and memory-intensive.
• Lack of Inherent Meaning: Individual characters carry very little semantic information. The model has the much harder task of learning not only the meaning of words but also how to compose them from characters in the first place.

The following video provides a concise summary of this trade-off.

3. Subword-Level Tokenization: A Pragmatic Compromise

Modern large language models use an intermediate approach called subword tokenization. The core idea is to balance the trade-offs between word-level and character-level methods.

• Commonly used words (like "the", "and", "cat") are kept as single, whole-word tokens.
• Rare or complex words (like "tokenization") are broken down into smaller, meaningful sub-units (e.g., token, ##ization, where ## indicates the start of a subword piece).
• Truly unknown words (like a complex chemical name or a new internet slang) can be broken down all the way to individual characters if necessary.

This approach offers the best of both worlds:

• Manages Vocabulary Size: The vocabulary size is a fixed hyperparameter (e.g., 32,000 or 50,000), making it much more manageable than a full word-level vocabulary.
• Eliminates OOV: Any word can be constructed from the subword and character units in the vocabulary. The <UNK> token is no longer necessary.
• Efficient Sequence Length: Sequences are longer than in word-level tokenization but significantly shorter than in character-level, striking a good balance for computational efficiency.
• Generalization: The model can learn meanings for subwords. If it learns what the suffix "-ly" means from words like "quickly" and "slowly," it can generalize that understanding to a new word like "efficiently."

The following reading provides a formal comparison of these three levels.

Algorithms like Byte Pair Encoding (BPE), WordPiece, and Unigram are used to learn an optimal subword vocabulary from a large text corpus. They do this by starting with characters and iteratively merging frequent pairs of tokens to build up the vocabulary of subwords.

Visualizing the Trade-offs

The differences between these approaches become stark when we compare the number of tokens they produce for various inputs.

A bar chart comparing the number of tokens generated by character-level, BPE (a subword method), and word-level tokenizers for different text samples.

Let's analyze the chart:

• "the cat sat": For common words, word-level and BPE are the most efficient, producing only 3 tokens. Character-level is highly inefficient.
• "computational": A more complex but regular word. Word-level is a single token. BPE breaks it into a few subwords (e.g., comput, ation, al). Character-level produces 13 tokens.
• "antidisestablishment": A rare, morphologically complex word. A word-level tokenizer would almost certainly map this to a single <UNK> token, losing all information. BPE gracefully handles it by splitting it into meaningful morphemes (anti, dis, establish, ment). Character-level tokenization results in a very long sequence of 20 tokens.
• "COVID-19 test": A modern neologism. Again, a word-level tokenizer trained before 2020 would fail on "COVID-19". BPE can represent it by combining known subwords and characters (COV, ID, -, 19).

This comparison clearly illustrates why subword tokenization has become the standard for virtually all modern large language models. It provides a robust and efficient method for converting any text into a sequence of tokens without the brittleness of word-level models or the extreme computational cost of character-level models.

Conclusion

In this lesson, we have dissected the crucial first step in processing text: tokenization. We compared the three main strategies and their inherent trade-offs:

• Word-level: Creates semantically rich tokens and short sequences but suffers from massive vocabularies and an inability to handle novel words (the OOV problem).
• Character-level: Solves the OOV problem with a tiny vocabulary but results in extremely long, computationally expensive sequences where semantic meaning is initially lost.
• Subword-level: Provides a pragmatic and powerful compromise. It uses a manageable vocabulary size, can represent any word, and produces sequences of a reasonable length, making it the de facto standard for state-of-the-art models.

We have now established what we are creating numerical representations for—these subword tokens. However, we've only touched on how a subword vocabulary is created. In the next lesson, we will dive into the mechanics of the most common subword algorithm, Byte Pair Encoding (BPE), and walk through the step-by-step process of how it builds a vocabulary from a raw text corpus.