Text to Tokens: The LLM's Language

Introduction

In our previous lesson, we established why the Transformer architecture marked a radical improvement over sequential models like RNNs. Its ability to process all input elements in parallel was the key that unlocked the scaling of models to today's massive sizes. This naturally leads to a crucial question: how do we get from a raw string of text to the numerical sequence that a Transformer can actually process?

This lesson will answer that question by diving into tokenization, the foundational step that converts human-readable text into a numerical format suitable for a Large Language Model. We will explore why simple word- or character-based approaches are insufficient and then focus on the mechanics of subword tokenization, specifically the Byte-Pair Encoding (BPE) algorithm used by models like GPT. Understanding tokenization is not just an academic exercise; it has direct implications for model behavior, context window limitations, and even the cost of using LLM APIs.

What are Tokens?

At its core, an LLM is a machine for predicting the next number in a sequence. The entire process begins by converting your input text, like "Hello world", into a series of numbers. These numbers are called Token IDs, and each one represents a piece of text called a token.

To get a practical sense of what this means and why it matters for developers, let's start with a short, engaging video from Matt Pocock.

As the video illustrates, a token isn't always a full word. It can be a word, a part of a word (a "subword"), a punctuation mark, or even a single character. The process of breaking text down into these units is called tokenization.

The Tokenization Trilemma: Words, Characters, or Subwords?

To build an intuition for why modern LLMs use subwords, let's consider the trade-offs between different tokenization strategies. This is a classic engineering problem of balancing competing constraints.

Let's summarize the key points:

1. Word-based Tokenization:

   • How it works: Split text by spaces and punctuation. ["The", "quick", "brown", "fox", "."]
   • Problem: This leads to an enormous vocabulary (every single word in a language). More importantly, it suffers from the out-of-vocabulary (OOV) problem. The model has no way to handle a word it didn't see during training, like a new name, a technical term, or a typo.

2. Character-based Tokenization:

   • How it works: Split text into individual characters. ["T", "h", "e", " ", "q", "u", "i", "c", "k", ...]
   • Advantage: The vocabulary is tiny (e.g., ~256 for all byte values), and there is no OOV problem. Any word can be constructed from characters.
   • Problem: Sequences become very long, making computation expensive. More fundamentally, a single character carries very little semantic meaning, making it much harder for the model to learn meaningful patterns in the language.

3. Subword Tokenization:

   • How it works: This is the hybrid solution used by most modern LLMs. It keeps frequent words as single tokens (e.g., "the", "and") but breaks down rarer words into smaller, meaningful sub-units (e.g., "tokenization" might become ["token", "ization"]).
   • Advantage: It strikes a balance. The vocabulary size is manageable, and it can represent any word by composing it from subwords, effectively eliminating the OOV problem.

This image neatly illustrates how a subword tokenizer can handle an unknown or rare word by breaking it into smaller pieces that are likely present in its vocabulary.

The most common algorithm for achieving this is called Byte-Pair Encoding (BPE).

How Byte-Pair Encoding (BPE) Works

BPE was originally developed as a data compression algorithm. Its application to NLP is a clever adaptation. The core idea is to iteratively merge the most frequent pairs of characters or character sequences in a training text to build up a vocabulary of common subwords.

For a detailed walkthrough, we'll turn to an excellent article by Sebastian Raschka that implements BPE from scratch. Given your background, you'll appreciate the algorithmic clarity.

To summarize the BPE training process:

1. Initialization: Start with a base vocabulary consisting of all individual characters (or bytes) present in the training data.
2. Iteration:
   a. Count all adjacent pairs of tokens in the corpus.
   b. Identify the most frequent pair (e.g., ('t', 'h')).
   c. Merge this pair into a new, single token (e.g., 'th').
   d. Add this new token to the vocabulary.
   e. Replace all occurrences of the original pair in the corpus with the new merged token.
3. Termination: Repeat this process for a specified number of merges. This number is a hyperparameter that determines the final vocabulary size (e.g., 50,257 for GPT-2).

The output of this training is a final vocabulary of tokens (from single characters to full words) and a list of the merge rules in the order they were learned. To encode a new piece of text, the tokenizer applies these learned merge rules to break the text down into the longest possible tokens found in its vocabulary.

The Full Picture: From Text to Token IDs

Now we can assemble the full picture. Tokenization is the first critical stage in the LLM input pipeline.

1. Input Text: A raw string, e.g., "This is an example."
2. Tokenized Text: The string is broken into subword tokens using the learned BPE vocabulary. e.g., ["This", " is", " an", " example", "."]
3. Token IDs: Each token is replaced by its unique integer ID from the vocabulary lookup table. e.g., [40134, 2052, 133, 389, 12]

This sequence of integers is the final output of the tokenizer.

This integer sequence is what the model receives. It is the language of the machine.

Conclusion

In this lesson, we've dissected the crucial process of tokenization. We saw how it acts as the bridge between human language and the numerical world of machine learning models. You learned about the fundamental trade-offs between word, character, and subword tokenization, and why the hybrid subword approach, exemplified by Byte-Pair Encoding (BPE), has become the standard for modern LLMs. The key takeaway is that BPE creates a compact and efficient vocabulary by iteratively merging frequent character pairs, allowing it to handle any text while balancing vocabulary size and sequence length.

This process leaves us with a sequence of Token IDs. But these are just arbitrary numbers. How does the model derive meaning from them? In our next lesson, we will explore the concepts of embeddings and self-attention. You'll learn how each Token ID is mapped to a rich, high-dimensional vector (an embedding) that captures its semantic meaning, and how the self-attention mechanism operates on these vectors to understand the contextual relationships between every token in a sequence.