Skip to main content
Back to path

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 32

Building a BPE Tokenizer

Recommended resources

Introduction

Welcome back! In our previous lesson, we established the theoretical foundations of tokenization, exploring why modern LLMs rely on subword methods and comparing the core ideas behind BPE, WordPiece, and SentencePiece. We learned that these algorithms are the gatekeepers that transform raw text into the numerical sequences models actually process.

Today, we transition from theory to practice. This lesson is designed to meet the learning outcome: Implement a basic Byte-Pair Encoding (BPE) tokenizer to understand vocabulary construction and subword segmentation. We will demystify the BPE algorithm by first walking through its logic step-by-step and then examining a clean, "from scratch" implementation. Finally, we'll see how to leverage powerful libraries like Hugging Face tokenizers and tiktoken to train and use BPE tokenizers in practice. This hands-on approach will solidify your understanding of how a model's vocabulary is born and how it deconstructs language.

The BPE Algorithm: A Conceptual Walkthrough

At its heart, the BPE algorithm used in NLP is a beautifully simple, greedy algorithm. It starts with a base vocabulary of individual characters and iteratively builds a larger vocabulary of subwords by merging the most frequently occurring adjacent pairs.

To see how this works in detail, we'll watch a video that provides an excellent whiteboard explanation. It starts by contrasting different tokenization strategies and then dives into a step-by-step BPE implementation on a small, example corpus.

Lecture 8: The GPT Tokenizer: Byte Pair Encoding

The video 'Lecture 8: The GPT Tokenizer: Byte Pair Encoding' by Vizuara provides one of the clearest conceptual walkthroughs of the BPE algorithm. We will focus on the sections where the algorithm is explained and applied to a sample vocabulary.

Please watch the segments from 02:48 to 38:58. As you watch, focus on understanding the following process: The Trade-offs (02:48 - 15:56): A great review of why subword tokenization (like BPE) is a good compromise between word-based and character-based methods. The Core Idea (15:56 - 21:39): How BPE, originally a data compression algorithm, is adapted for tokenization by finding and merging frequent byte pairs. The Training Process (21:39 - 38:58): Pay close attention to the example with the words old, older, finest, lowest. Observe how the algorithm: Starts with individual characters. Identifies the most frequent pair (e.g., e and s). Merges them into a new token (es). Updates frequencies and repeats the process, building up meaningful subwords like est and old. Decides when to stop merging.

This process of identifying and merging pairs is how the tokenizer learns its vocabulary and merge rules from a large training corpus. The number of merges is a key hyperparameter that determines the final vocabulary size.

BPE Tokenization Process Illustration
This image illustrates the BPE process. An input word like "unbelievable" is first split into characters. Then, based on learned merge rules, these characters are iteratively combined into meaningful subwords ("un", "believ", "able"). Finally, these subwords are mapped to their corresponding integer IDs from the vocabulary.

Implementing BPE From the Ground Up

To truly master a concept, it's invaluable to see its implementation in clean, readable code. Andrej Karpathy's minbpe repository provides a minimal, educational implementation of the BPE algorithm. Studying this will connect the conceptual steps from the video to concrete Python code.

Similarly, Sebastian Raschka provides an excellent tutorial that builds a BPE tokenizer from scratch, explaining each component along the way. We'll use both of these resources to dissect the algorithm.

karpathy/minbpe: Minimal, clean code for the Byte Pair ...

First, let's read the project description for minbpe to understand its purpose and structure. This will set the context for the code we'll be examining.

Read the main README file, focusing on the 'Repository files navigation', 'quick start', and 'training' sections. Note the distinction between BasicTokenizer (simpler) and RegexTokenizer (more advanced, used in GPT models). We will focus on the BasicTokenizer logic.

Now, let's dive into a more detailed article that not only describes the algorithm but also provides a self-contained Python class for it. This will be our main guide for understanding the implementation.

Implementing A Byte Pair Encoding (BPE) Tokenizer From ...

Sebastian Raschka's blog post 'Implementing A Byte Pair Encoding (BPE) Tokenizer From ...' walks through the theory and provides a clear, well-commented Python class that implements BPE. This is an ideal resource for understanding the code.

Read the sections 'BPE algorithm outline,' 'BPE algorithm example,' and 'BPE implementation walkthrough.' As you read, try to map the code in his BPETokenizerSimple class (which you can see in the full article) to these core functions: A function to get pair statistics: This corresponds to the get_stats method in Karpathy's BasicTokenizer. The main training loop: This loop iterates for vocab_size - 256 times, finding the best pair and merging it in each step. A merge function: This replaces occurrences of a pair with a new token ID. Look for the merge function in Karpathy's code. Studying these components will reveal the full mechanics of both training (train) and inference (encode).

The core logic of the encode method, which tokenizes new text using the learned merges, is to repeatedly apply the merge rules in the order they were learned (from highest to lowest priority) until no more merges can be made.

Test your understanding!

A BPE tokenizer is trained on a corpus, resulting in the following two merge rules (in order of priority):

  1. Merge ('t', 'h') into token ID 256.
  2. Merge (256, 'e') into token ID 257.

How would the string "the" be tokenized by the encode method? Trace the steps.

Show answer

  1. Initial state: The string "the" is first converted into a sequence of its byte/character IDs. Let's assume t=116, h=104, e=101. The sequence is [116, 104, 101].

  2. Apply Merge 1: The encode function searches for the highest-priority merge pair, ('t', 'h') or (116, 104). It finds one occurrence.

    • The sequence is updated by replacing [116, 104] with [256].
    • The new sequence is [256, 101].

  3. Apply Merge 2: The function now restarts its search from the highest-priority merge rule on the new sequence.

    • It checks for ('t', 'h') again. There are no more occurrences.
    • It checks for the next rule, (256, 'e') or (256, 101). It finds one occurrence.
    • The sequence is updated by replacing [256, 101] with [257].

  4. Final Result: The final tokenized sequence is [257]. The word "the" has been successfully represented by a single token.

Practical BPE: Training and Using Tokenizers

While building a tokenizer from scratch is insightful, in real-world projects, you'll use highly optimized libraries. The two most common are Hugging Face's tokenizers for training custom tokenizers and OpenAI's tiktoken for using their pre-trained ones.

Training Your Own BPE Tokenizer with Hugging Face

Let's say you're working in a specialized domain like finance or medicine. A pre-trained tokenizer might be inefficient, splitting common domain terms into many subwords. Training your own tokenizer on domain-specific text can significantly improve performance and efficiency.

The tokenizers library makes this process straightforward.

Python code to build your BPE - Tokenizer from scratch (w/ HuggingFace)

The video 'Python code to build your BPE - Tokenizer from scratch (w/ HuggingFace)' by Discover AI demonstrates how to use the tokenizers library to train a BPE model on a custom text corpus.

Watch the segment from 03:05 to 07:31. Focus on these key steps: Instantiation (03:05 - 04:11): How to import and instantiate a BPE model and a BpeTrainer, specifying parameters like vocab_size. Training (04:34 - 05:30): The simple tokenizer.train() method that takes your text files and the trainer object. Application and Analysis (05:30 - 07:31): How the newly trained tokenizer is used to encode a sentence. Note the crucial point made: if the training data is mismatched with the target text (e.g., training on Wikipedia but encoding quantum mechanics), the tokenization will be poor.

Using a Pre-trained BPE Tokenizer with tiktoken

More often, you will use an existing, pre-trained tokenizer that comes with a powerful model like GPT-4 or Llama 3. The tiktoken library is OpenAI's high-performance tokenizer library, which gives you access to the exact tokenizers used by their models.

Let's return to the first video, which also demonstrates how to use tiktoken.

Lecture 8: The GPT Tokenizer: Byte Pair Encoding

We'll now watch the final part of the Vizuara video, which provides a hands-on demonstration of using tiktoken to load and use the GPT-2 tokenizer.

Watch the segment from 40:53 to 51:18. Pay attention to: Loading a Tokenizer: How tiktoken.get_encoding("gpt2") instantly gives you the pre-trained tokenizer. Encode/Decode: The simple .encode() and .decode() methods. Handling OOV words: Notice how the BPE tokenizer gracefully handles a nonsensical string like "a k w i r w i e r" by breaking it down into individual characters and subwords it knows, completely avoiding an 'unknown token' error. This demonstrates the robustness we discussed in the last lesson.

Conclusion

Today you bridged the gap between the theory and practice of Byte-Pair Encoding. By dissecting the algorithm's logic and exploring its implementation from scratch, you've gained a deep appreciation for the mechanics of vocabulary construction and subword segmentation.

Key Takeaways:

  • BPE Training is a Greedy Merge Process: The algorithm starts with characters and iteratively merges the most frequent adjacent pair to build up a subword vocabulary.
  • Implementation from Scratch: The core logic involves a loop that gets pair statistics, identifies the best pair, and merges it, updating a list of merge rules.
  • The encode Method Applies Merges: To tokenize new text, the learned merge rules are applied in order of priority to the sequence of input characters.
  • Practical Tooling: In practice, we use libraries like Hugging Face tokenizers to train custom BPE models on domain-specific data and tiktoken to use the high-performance, pre-trained tokenizers from models like GPT.

What's Next?

We've now mastered the first critical step of the LLM training pipeline: converting raw text into tokens. But this raises fundamental questions for training a model: How much data do we need? And how large should the model be to effectively learn from that data?

In the next lesson, we will explore Scaling Laws. These are empirical principles, most famously studied by OpenAI, that describe the mathematical relationship between model size, dataset size, and a model's final performance. Understanding these laws is crucial for making informed decisions when training or selecting a model for a specific task.