Word Embeddings and Transformer Architecture in NLP

1. From Words to Vectors: The Idea of an Embedding

Before a neural network can process text, the words must be converted into numbers. A naive approach, like one-hot encoding (representing each word as a long vector of zeros with a single '1'), is deeply flawed. It creates astronomically large vectors and, more importantly, treats every word as being equally different from every other. "Dog" is as distant from "canine" as it is from "democracy".

To solve this, NLP uses word embeddings. The core idea is to represent each word as a relatively short, dense vector (a list of numbers) in a high-dimensional space. The key principle is that words with similar meanings should have vectors that are close to each other in this space.

These embeddings are not hand-crafted; they are learned. During training on a vast corpus of text, the network adjusts the vector for each word, positioning it in this "semantic space" based on the contexts in which it appears. This process can capture surprisingly rich relationships. The most famous example is the vector arithmetic: vector('King') - vector('Man') + vector('Woman') results in a vector very close to vector('Queen').

The following video provides an excellent visual introduction to this concept.

The embedding matrix, which stores the vector for every word in the model's vocabulary, constitutes the very first set of learnable parameters in our model.

2. The Rise of Context: Introducing the Transformer

Static embeddings like the ones we just saw were a major breakthrough, but they have a crucial limitation: a word has the same embedding regardless of its context. As noted earlier, this fails to distinguish "river bank" from "investment bank". To truly understand language, a model must be able to create contextualized representations.

This is the problem that the Transformer architecture, introduced in the 2017 paper "Attention Is All You Need," was designed to solve. It has since become the foundation for virtually all state-of-the-art LLMs.

At a high level, a Transformer processes a sequence of token embeddings and, through a series of "Transformer Blocks," outputs a new sequence of embeddings where each vector has been enriched with contextual information from the entire input.

Let's watch a high-level overview of how data flows through a generative Transformer.

3. The Core Mechanism: Self-Attention

The magic of the Transformer lies in its self-attention mechanism. This is how the model weighs the relevance of all other words in the input when interpreting a specific word. For example, when processing the word it in the sentence "The car stalled because it ran out of fuel," the attention mechanism should learn to pay high attention to car and fuel.

To accomplish this, self-attention projects each input word's embedding into three distinct vectors:

• Query (Q): Represents the current word's "question" or what it's looking for. Think of it as a search query.
• Key (K): Represents each word's "label" or what kind of information it offers. Think of it as the title of a webpage.
• Value (V): Represents the actual content or meaning of each word. Think of it as the content of the webpage.

The process then unfolds as follows:

1. Calculate Attention Scores: For the current word, its Query vector is compared with every other word's Key vector using a dot product. A high score means a strong relevance.
2. Normalize Scores: These raw scores are passed through a softmax function, which converts them into a set of positive weights that sum to 1. These are the attention weights. They represent a probability distribution of "focus" over the input sequence.
3. Produce the Output: The final output for the current word is calculated as a weighted sum of all the Value vectors in the sequence, using the attention weights from the previous step.

In this way, a word's representation is updated to be a mix of its own meaning and the meanings of the words to which it "attends".

The next video in the 3Blue1Brown series provides an exceptional step-by-step visual walkthrough of this mechanism.

Multi-Head Attention

A Transformer doesn't just do this once. It uses Multi-Head Attention, which means it runs the self-attention process multiple times in parallel, each with its own set of learned Query, Key, and Value weight matrices. Each "head" can learn to focus on different types of relationships. For instance, one head might track syntactic dependencies (like subject-verb agreement), while another might track semantic relationships (like coreference).

The outputs from all heads are then concatenated and passed through another linear layer to produce the final output of the multi-head attention block.

4. The Full Picture: Transformer Blocks and Prediction

The Multi-Head Attention mechanism is the heart of a Transformer Block. This block also contains two other key components that we've seen before:

• A Feed-Forward Network (FFN): A standard two-layer fully-connected network that is applied independently to each token's representation after attention. Its role is to add further processing and representational capacity.
• Residual Connections and Layer Normalization: These are crucial for enabling stable training of very deep networks (e.g., GPT-3 has 96 stacked Transformer blocks). The residual connection adds the input of a layer to its output, helping to prevent the "vanishing gradient" problem.

To generate text, the sequence of input tokens is processed through the entire stack of Transformer blocks. Finally, the output embedding of the very last token is taken and passed through a final linear layer (often called the "unembedding" layer) and a softmax function. This produces a probability distribution over the entire vocabulary for what the next token should be. The model then samples from this distribution to pick the next word, appends it to the sequence, and repeats the entire process.

For a comprehensive, formal text-based summary of everything we've covered—from embeddings to the full transformer block and the final language modeling head—the following resource is an excellent reference.

Finally, to see all these pieces work together in a dynamic way, I highly recommend exploring this interactive tool.

Conclusion

In this lesson, we have journeyed into the core of modern natural language processing. We've moved beyond the static world of simple word vectors into the dynamic, context-aware realm of the Transformer.

Key Takeaways:

• Word embeddings are dense vector representations of words that capture semantic relationships, with similar words being closer in the vector space.
• The Transformer architecture was designed to create contextualized word representations by processing an entire sequence at once.
• Its core innovation is the self-attention mechanism, which allows the model to weigh the importance of other words in the sequence when processing a given word.
• This mechanism is powered by learnable Query, Key, and Value projections for each word.
• Multi-Head Attention allows the model to learn different kinds of linguistic relationships in parallel.
• A full Transformer consists of stacked Transformer Blocks (Attention + FFN), culminating in a prediction head that generates a probability distribution for the next word.

Preview of the Next Lesson:

We have now explained the 'T' in GPT (Generative Pre-trained Transformer). But what about the "Large" in Large Language Models? In our next lesson, we will describe the significance of the Transformer model for the development of LLMs. We will explore how the parallelizable nature of this architecture unlocked unprecedented scale, leading to the powerful and sometimes unpredictable models that are at the center of AI governance debates today.