Tokenizing Text for Sequence Models

Introduction

Welcome back! In our last lesson, we delved into the architectures of LSTMs and GRUs, understanding how their gating mechanisms solve the vanishing gradient problem and enable them to learn long-range dependencies. These models are powerful tools for processing sequences.

However, a fundamental question remains: how do we feed a sequence of words, like a sentence, into a neural network that only understands numbers? This lesson bridges that gap. We will focus on tokenization, the essential preprocessing step that converts raw text into a numerical format suitable for sequence models like LSTMs.

By the end of this 60-minute lesson, you will be able to preprocess text data by:

• Understanding the core concepts of tokenization, vocabularies, and text normalization.
• Comparing different tokenization strategies, including word, character, and subword-level approaches.
• Recognizing the critical trade-off between vocabulary size and sequence length.
• Using torchtext to build a vocabulary and tokenize a batch of text, complete with special tokens for padding and out-of-vocabulary words.

The Core Idea: From Text to Numbers

At its heart, tokenization is the process of breaking down a piece of text into smaller units called tokens and then mapping these tokens to unique integer IDs. These IDs are what the neural network will actually process.

The overall process can be visualized as a pipeline:

The central component of this pipeline is the vocabulary, which is essentially a dictionary that maps each unique token to a specific integer.

Let's briefly outline the steps involved:

1. Normalization: The text is cleaned. This typically involves converting all text to lowercase, removing unwanted characters, and handling punctuation.
2. Splitting: The normalized text is split into a list of tokens. The simplest method is to split by spaces to get word tokens.
3. Vocabulary Building: We scan a large body of text (a corpus) to find all unique tokens and assign a unique integer ID to each one.
4. Encoding: Using the vocabulary, we convert a new sentence into a sequence of integer IDs.

To build an intuition for the data structures involved, the PyTorch chatbot tutorial defines a Voc class from scratch. This class manages the mappings between words and indices (word2index, index2word) and counts word frequencies. You don't need to implement this yourself, as modern libraries handle it, but it's helpful to see the logic.

Tokenization Strategies and Their Trade-offs

How we choose to split text into tokens is a critical decision with significant consequences for model performance and efficiency. There are three main strategies.

To get a clear overview of these strategies, let's watch a short video.

As the video explained, each method has a distinct profile:

1. Word Tokenization:

   • How it works: Splits text by spaces or punctuation.
   • Pros: Intuitive. Creates relatively short sequences. Tokens (words) are semantically meaningful.
   • Cons: Can result in a massive vocabulary (e.g., >100,000 words for English). It suffers from the Out-of-Vocabulary (OOV) problem: it cannot handle words that were not in the training data.

2. Character Tokenization:

   • How it works: Splits text into individual characters.
   • Pros: The vocabulary is tiny (e.g., ~100 characters for English), and there is no OOV problem.
   • Cons: Produces very long sequences, which are computationally expensive to process. Individual characters carry little semantic meaning, making it harder for the model to learn word-level concepts.

3. Subword Tokenization (e.g., Byte-Pair Encoding, BPE):

   • How it works: An adaptive algorithm that learns to merge frequent pairs of characters. The result is that common words become single tokens, while rare words are broken down into smaller, meaningful subword units (e.g., "tokenization" -> ["token", "ization"]).
   • Pros: Strikes a balance. It keeps the vocabulary a manageable size (typically 30k-50k), produces reasonably short sequences, and can handle any word by breaking it down into known subwords, effectively solving the OOV problem.
   • Cons: The tokens are less interpretable to humans than full words.

The Fundamental Trade-off

This leads us to the fundamental trade-off in tokenization: vocabulary size versus sequence length. Your background in data structures and algorithms makes this a familiar concept of space-time trade-offs.

• Small Vocabulary (e.g., Characters): Low memory for embeddings, but high computational cost during training/inference due to long sequences (attention complexity is quadratic in sequence length, $O(L^2)$).
• Large Vocabulary (e.g., Words/Subwords): Higher memory for embeddings, but lower computational cost due to shorter sequences.

Modern language models almost universally use subword tokenization because the reduction in sequence processing cost far outweighs the increased memory cost of the embedding table.

Practical Tokenization with torchtext

Now that we understand the theory, let's see how to implement tokenization using PyTorch's torchtext library. We'll follow a practical example of preparing the AG News dataset for a classification task.

The process involves three main steps:

1. Defining a tokenizer to split sentences.
2. Building a vocabulary from our training data.
3. Creating a processing pipeline to convert batches of text into padded numerical tensors.

Let's summarize the key takeaways from the video:

• Tokenizer: torchtext.data.utils.get_tokenizer('basic_english') provides a simple, effective word-level tokenizer that lowercases text and splits on spaces and punctuation.
• Vocabulary: torchtext.vocab.build_vocab_from_iterator is a powerful function. It takes an iterator that yields lists of tokens and automatically builds the word-to-ID mapping. You can specify a min_freq to discard rare words and add specials like <unk> and <pad>.
• Special Tokens:
  • <unk> (Unknown): The token used for any word not present in the vocabulary. This gracefully handles the OOV problem in word-level tokenization.
  • <pad> (Padding): Used to fill shorter sequences in a batch so that all sequences have the same length. This is necessary for creating rectangular tensors that can be processed efficiently on GPUs.
  • <sos> / <eos> (Start/End of Sentence): Optional tokens that can mark the beginning and end of a sequence, which is particularly useful in sequence-to-sequence tasks like translation.
• Pipeline: The process of applying these steps (tokenizing, numericalizing, padding) can be chained together. The video shows this by applying transforms to a DataLoader. The final output is a tensor of shape (sequence_length, batch_size), which is the standard input format for RNNs in PyTorch, as it allows the model to process one time step at a time across the entire batch.

Conclusion

In this lesson, we demystified the crucial first step of any NLP pipeline: converting raw text into numbers. You've moved from the theory of LSTMs to the practicalities of preparing their input data.

Key Takeaways:

• Tokenization is the process of splitting text into tokens (words, characters, or subwords) and mapping them to integer IDs using a vocabulary.
• The choice of tokenization strategy involves a critical trade-off between vocabulary size and sequence length.
• Subword tokenization (like BPE) is the modern standard, as it balances vocabulary size, sequence length, and robustly handles unknown words.
• Special tokens like <pad> and <unk> are essential for handling batches of variable-length sequences and out-of-vocabulary words.
• Libraries like torchtext provide high-level utilities to build vocabularies and create tokenization pipelines efficiently.

Now that we can convert text into sequences of integers, a new question arises. A 50,000-ID vocabulary would mean our integers range from 0 to 49,999. Should we feed these directly to the model? Or maybe use a one-hot encoding? Both are problematic due to the lack of semantic relationship and extreme sparsity.

In our next lesson, we will explore embedding layers, the standard and powerful solution to this problem. You will learn how these layers map token IDs to dense, low-dimensional vectors that capture semantic relationships between words, providing a rich and efficient input representation for our sequence models.