Tokenization and Embeddings

Hello! Welcome to the third lesson in our journey from neural networks to modern LLMs.

In the previous lesson, we demystified how a neural network "learns" by iteratively adjusting its internal parameters (weights and biases) to minimize a loss function, using the powerful combination of gradient descent and backpropagation. We established that this entire process operates on numbers.

This brings us to a fundamental question: How do we represent human language as numbers so a neural network can process it? Computers don't understand words, they understand numerical data. This lesson will answer that question, addressing the learning outcome: Explain how text is converted into tokens and token embeddings.

We will explore this crucial two-step process:

1. Tokenization: Breaking down raw text into smaller, manageable pieces called tokens.
2. Embedding: Converting these tokens into rich, numerical vectors that capture their semantic meaning.

Let's begin.

1. Tokenization: The Language of LLMs

Large Language Models don't see text as words or characters. Instead, they operate on tokens. Tokenization is the first step in any NLP pipeline, where we convert a string of text into a sequence of tokens. A token can be a word, a part of a word (a subword), or even a single character.

To get a quick overview of this process, let's watch a short segment from LearnThatStack.

As the video explained, this conversion is handled by a tokenizer. Every LLM has an associated tokenizer that was "trained" on a massive text corpus to build a vocabulary—a list of all the tokens it knows. Each unique token in this vocabulary is mapped to a unique integer ID.

So, the process is:
"I love programming" → ["I", "love", "programming"] (Tokens) → [40, 831, 11502] (Token IDs)

It is these integer IDs, not the text itself, that are fed into the neural network.

Why Not Just Use Words? Or Characters?

You might wonder why we need this special tokenization process.

• Word-level tokenization would lead to an enormous vocabulary (all words in a language, plus names, slang, misspellings). This makes the model huge and slow. It also struggles with words it has never seen before (out-of-vocabulary or OOV words).
• Character-level tokenization creates a very small vocabulary (e.g., A-Z, a-z, 0-9, punctuation). However, it results in very long sequences of tokens, and it forces the model to learn the concept of words from scratch, which is computationally inefficient.

The solution used by almost all modern LLMs is subword tokenization.

Subword Tokenization: The Best of Both Worlds

Subword tokenization strikes a balance.

• Common words like "the" or "and" are kept as single tokens.
• Rare or complex words are broken down into smaller, meaningful subwords. For example, "tokenization" might become token + ization.

This approach keeps the vocabulary size manageable while allowing the model to understand and construct a virtually unlimited number of words. A popular algorithm for creating these subword vocabularies is Byte-Pair Encoding (BPE). BPE starts with a vocabulary of individual characters and iteratively merges the most frequently occurring adjacent pairs of tokens.

Given your background in software engineering, you might appreciate seeing how this works more concretely. The following video demonstrates tokenization using OpenAI's tiktoken library.

2. Token Embeddings: Adding Meaning to Numbers

We now have a sequence of integer IDs. However, these numbers are arbitrary. The ID for "king" (e.g., 2345) has no mathematical relationship to the ID for "queen" (e.g., 5432). The model needs a way to understand that "king" is semantically closer to "queen" than it is to "car".

This is the job of the embedding layer.

An embedding is a dense vector (a list of floating-point numbers) that represents a token's meaning. Each token ID is mapped to a unique, high-dimensional vector. For example, a model might represent every token with a vector of length 768.

This is achieved by using an embedding table, which is essentially a large lookup table (a matrix). The token ID serves as the index to retrieve the corresponding vector from this table.

[40, 831, 11502] → [[0.1, -0.5, ...], [-0.2, 0.8, ...], [0.9, 0.1, ...]]

Let's return to the first video to see a great visualization of this concept.

The magic of embeddings lies in the fact that these vectors are not random or fixed. They are learned parameters. Just like the weights and biases in the rest of the network, the values in the embedding table are adjusted during training via backpropagation. The model learns to place vectors for semantically similar tokens close together in this "meaning space" because doing so helps it better predict the next token, thus minimizing the loss function.

This leads to the famous analogy: vector('king') - vector('man') + vector('woman') results in a vector that is very close to vector('queen').

Distinguishing Tokenization and Embeddings

It is critical to understand the difference between these two stages. They are complementary but distinct.

To summarize the distinction from the article:

• Tokenization is a structural, pre-processing step. It breaks text into discrete integer IDs based on a fixed vocabulary.
• Embedding is a semantic, learned step. It converts those discrete IDs into continuous, meaningful vector representations that the model can perform mathematical operations on.

Here is a visual summary of the entire input pipeline:

Conclusion

In this lesson, we've built the bridge from human language to the numerical world of neural networks. You now understand the essential two-step process that prepares text for an LLM.

Key Takeaways:

• Tokenization is the process of breaking text into a sequence of tokens (words, subwords, or characters) and converting them into integer IDs using a pre-defined vocabulary.
• Subword tokenization (e.g., BPE) is the standard method, balancing vocabulary size and sequence length.
• Embeddings are learned vector representations of tokens. They map arbitrary token IDs to a high-dimensional "semantic space" where distance corresponds to similarity in meaning.
• The embedding vectors are parameters that the model learns during training, just like its other weights and biases.

We've successfully converted our text into a sequence of meaningful vectors. But there's a missing piece. A sentence is an ordered sequence of words, but the process we've described so far discards this order. How does the model know that "the dog chased the cat" is different from "the cat chased the dog"?

In our next lesson, we will explore positional encoding, the clever mechanism that re-introduces this crucial sequential information to the model.