Transformer Architecture and Self-Attention

1. From Sequences to Parallelism: The Rise of the Transformer

Before 2017, the dominant architectures for handling sequential data like text were Recurrent Neural Networks (RNNs) and their variants like LSTMs. While powerful, they had a fundamental limitation: they process data sequentially, word by word. This makes it difficult to parallelize computations and can lead to challenges in capturing long-range dependencies in text—for example, connecting a pronoun at the end of a long paragraph to the noun it refers to at the beginning.

The Transformer, introduced in the paper "Attention Is All You Need," proposed a new architecture that processes all input tokens simultaneously. It relies entirely on a mechanism called self-attention to draw connections and understand relationships between words, no matter how far apart they are in the sequence.

At a high level, the original Transformer has two main parts: an Encoder and a Decoder.

• The Encoder's job is to read the input text (e.g., a sentence in one language) and build a rich, context-aware numerical representation of it.
• The Decoder's job is to take that representation and generate an output sequence (e.g., the translated sentence).

For your project's goal of complaint classification, you're not generating new text; you're understanding and categorizing existing text. Therefore, you'll primarily be interested in the Encoder part of the architecture, which is the foundation of models like BERT (Bidirectional Encoder Representations from Transformers).

The following article by Jay Alammar provides an exceptional visual and conceptual introduction to the Transformer architecture.

2. The Heart of the Transformer: Self-Attention

The magic of the Transformer lies in its self-attention mechanism. It allows the model to weigh the importance of different words in a sequence when producing a representation for a specific word.

Imagine the complaint: "The meal was served without dal because it was spoiled." For a human, it's obvious that "it" refers to "dal," not the "meal." Self-attention is the mechanism that allows the model to learn this association. As it processes the word "it," self-attention helps it "look back" at the other words in the sentence and determine that "dal" is the most relevant piece of context.

The 3Blue1Brown video below provides a fantastic, intuitive walkthrough of why attention is needed and how it refines a word's meaning based on its context.

3. The Mechanics of Self-Attention: Query, Key, and Value

So, how does self-attention work computationally? It's a series of matrix operations that can be broken down into a few key steps. For each input word's embedding, we create three distinct vectors: a Query, a Key, and a Value. These are generated by multiplying the embedding by three unique weight matrices (WQ, WK, WV) that are learned during training.

You can think of their roles analogously to a library search:

1. Query (Q): This is your search query. It represents the current word's request for information. For the word "it," the query might be, "What noun could I be referring to?"
2. Key (K): This is like the keywords on the spine of a book. Each word in the sentence generates a key, announcing what kind of information it holds. The word "dal" might have a key that says, "I am a food item, a noun."
3. Value (V): This is the content of the book itself. It's the actual meaning or information that a word is willing to share.

The process then unfolds as follows:

1. Score: For the word we are currently processing, we take its Query vector and compute the dot product with the Key vector of every other word in the sentence (including itself). A higher dot product means a better match—the "query" found a relevant "key."
2. Scale and Normalize (Softmax): The scores are scaled down (by dividing by the square root of the key vector's dimension, $\sqrt{d_k}$) to stabilize gradients. Then, a softmax function is applied to the scores. This converts them into a set of positive weights that sum to 1.0, effectively creating a probability distribution known as the attention weights. These weights determine how much focus to place on each word in the sentence.
3. Weighted Sum: Finally, we multiply these attention weights by each word's Value vector and sum them up. The result is a new vector, a context-aware representation for our original word. If "dal" had a high attention weight, its value vector contributes significantly to the new representation of "it," effectively encoding that "it" means "dal."

The following resources break this process down in detail, first with diagrams and then with corresponding code.

For a more code-oriented perspective, Sebastian Raschka's article walks through a from-scratch implementation in PyTorch, which directly relates to the tools you're using.

4. Multi-Head Attention and the Full Picture

A single self-attention mechanism is powerful, but it can be limiting. It forces the model to average all types of contextual information into one representation. Multi-Head Attention solves this by running the self-attention mechanism multiple times in parallel, each with a different, independently initialized set of Q, K, and V weight matrices.

Each of these parallel "attention heads" can learn to focus on different types of relationships. For example:

• One head might focus on syntactic relationships (e.g., which verb modifies which noun).
• Another head might focus on semantic relationships (e.g., which words are synonyms).
• Another might focus on positional relationships (e.g., which word comes right after another).

The outputs from all the attention heads are then concatenated and passed through a final linear layer. This gives the model a much richer, multi-faceted understanding of the input text.

Finally, to account for word order (which the self-attention mechanism itself ignores), the Transformer adds Positional Encodings to the initial input embeddings. These are vectors that follow a specific pattern, giving the model a signal about the position of each word in the sequence.

The StatQuest video below gives a great summary of how embeddings, positional encoding, and self-attention come together in an encoder-only model to create the final "context-aware embeddings" that you'll use for classification.

Conclusion

In this lesson, you've unpacked the fundamental architecture that powers most of modern NLP. You've gone from a high-level view of the Transformer down to the specific matrix calculations that drive its core mechanism, self-attention.

Here are the key concepts to take away:

• The Transformer architecture abandoned recurrence in favor of a parallelizable self-attention mechanism.
• Self-attention allows the model to weigh the importance of all words in a sequence when encoding a specific word, capturing rich contextual information.
• This mechanism is driven by Query, Key, and Value vectors, which are used to compute attention scores and create a weighted representation.
• Multi-Head Attention enhances this by allowing the model to learn different types of relationships simultaneously.
• Positional Encodings are added to the input to give the model information about word order.

You now have the necessary conceptual foundation to tackle your text classification task. In the next lesson, we will move from theory to practice. You will learn how to take a powerful, pre-trained Transformer encoder model—BERT—and fine-tune it on your dataset of parent complaints to build your second baseline classifier.