Tokenization Explained: From Text to Token IDs

Introduction

Today, we will address a fundamental question: How do we convert raw text, which is a sequence of characters, into a format that a neural network can process?

Just as a Convolutional Neural Network (CNN) operates on a grid of numerical pixel values, a language model needs a numerical representation of language. This lesson focuses on the very first step of that process: tokenization. Our goal is to understand how any piece of text is transformed into a sequence of integer numbers, known as token IDs. This is the bedrock upon which all modern Large Language Models (LLMs) are built.

Let's begin by looking at a high-level overview of where tokenization fits into the bigger picture.

The Core Problem: Text as Numbers

At their core, neural networks are mathematical functions that operate on tensors (multi-dimensional arrays) of numbers. They cannot directly understand strings like "hello" or "world". Therefore, our first challenge is to devise a systematic way to convert any text into a sequence of integers.

This process involves two main ideas:

1. Splitting: Breaking the text down into smaller chunks called tokens.
2. Mapping: Assigning a unique integer ID to each of these unique tokens using a dictionary, or vocabulary.

The seemingly simple question of "how should we split the text?" has profound implications for a model's performance, efficiency, and ability to handle language. Let's explore the primary strategies.

Tokenization Strategies: Words, Characters, and Subwords

There are three main ways to split text into tokens:

1. Word-based Tokenization: Each unique word in the text becomes a token.
2. Character-based Tokenization: Each unique character becomes a token.
3. Subword-based Tokenization: A hybrid approach where common words are kept whole, but rare words are broken down into smaller, meaningful pieces.

Each approach has significant trade-offs related to vocabulary size, the length of the resulting numerical sequence, and the ability to handle unknown words.

To get a clear understanding of these trade-offs, please watch the following video. It provides an excellent explanation of why subword tokenization has become the standard for modern LLMs like GPT.

As you saw, word and character tokenization represent two extremes. Word-based methods struggle with massive vocabularies and unknown words, while character-based methods create very long sequences and lose the semantic meaning of words. Subword tokenization offers a "Goldilocks" solution, and the most common algorithm used to achieve this is Byte-Pair Encoding (BPE).

How Byte-Pair Encoding (BPE) Works

BPE was originally a data compression algorithm. Its adaptation for NLP is what enables the powerful subword tokenization used in models like GPT-2, Qwen, and DeepSeek. The core idea is to start with individual characters and iteratively merge the most frequently occurring adjacent pairs of tokens.

Let's see how this works in practice.

This iterative merging process allows the tokenizer to learn a vocabulary that is highly efficient. It captures common morphemes (like est, er, ing) and frequently used words as single tokens, while still being able to represent any new or rare word by breaking it down into known subword or character pieces.

The Final Step: Token-to-ID Mapping

After the BPE algorithm has been trained on a large corpus of text, we are left with a fixed vocabulary – a list of all possible tokens (characters, subwords, and full words). The final step is to create a simple mapping, like a Python dictionary, that assigns a unique integer to each token in this vocabulary. This integer is the token ID.

The following resource provides a clear, concise explanation of this process, from building the vocabulary to the encode (text-to-ID) and decode (ID-to-text) functions.

Let's visualize the final mapping. An input string is first broken into tokens based on the learned BPE rules, and then each token is replaced by its corresponding integer ID from the vocabulary.

For a final, practical look, here is how you would use a real tokenizer from the transformers library to get token IDs.

from transformers import AutoTokenizer




# Load a pre-trained GPT-2 tokenizer
tokenizer = AutoTokenizer.from_pretrained("gpt2")

text = "Tokenization is fascinating!"




# 1. Split text into subword tokens
tokens = tokenizer.tokenize(text)
print(f"Tokens: {tokens}")



# Output: Tokens: ['Token', 'ization', 'Ġis', 'Ġfascinating', '!']
# Note: 'Ġ' represents a space character.




# 2. Convert tokens to their integer IDs
token_ids = tokenizer.convert_tokens_to_ids(tokens)
print(f"Token IDs: {token_ids}")



# Output: Token IDs: [11241, 4712, 318, 12691, 0]

This sequence of integers, [11241, 4712, 318, 12691, 0], is the final numerical representation of the input text that is fed into the model.

Conclusion

In this lesson, we have established the crucial first step in any language model pipeline.

Key Takeaways:

• Neural networks operate on numbers, so text must be converted into a numerical format.
• Tokenization is the process of splitting text into units (tokens). These can be words, characters, or, most effectively, subwords.
• Byte-Pair Encoding (BPE) is a common algorithm that learns an efficient subword vocabulary by iteratively merging the most frequent adjacent character pairs from a large text corpus.
• This subword approach effectively handles unknown words and balances the trade-off between vocabulary size and sequence length.
• The final step is mapping each token in the vocabulary to a unique integer token ID, creating the final sequence of numbers that the model processes.

Preview of the Next Lesson:

We now have a sequence of integer IDs. But these are just arbitrary indices. How does the model learn that the token for "car" is semantically closer to "truck" than to "apple"? In the next lesson, we will explore embedding layers, which solve this problem by mapping each token ID to a dense, meaningful vector representation.