In our previous lesson, we deconstructed the decoder-only architecture, the engine behind generative models like GPT. We saw how its unidirectional, autoregressive nature, enforced by causal self-attention, makes it perfectly suited for predicting the next token in a sequence. Now, we turn our attention to its architectural counterpart, designed not for generation, but for deep contextual understanding.
This lesson explores the encoder-only architecture, famously embodied by BERT (Bidirectional Encoder Representations from Transformers). We will dissect its structure, understand its bidirectional attention mechanism, and investigate the clever training objectives that allow it to "see" an entire sentence at once. By the end, you will understand why this design excels at natural language understanding (NLU) tasks like classification, sentiment analysis, and question answering.
The Encoder-Only Paradigm: A Focus on Representation
Just as we derived the decoder-only model by isolating the decoder stack of the original Transformer, we can derive the encoder-only model by discarding the decoder and keeping only the encoder stack. This creates a powerful model whose sole purpose is to take a sequence of tokens and produce a sequence of rich, contextualized numerical representations.
The fundamental difference lies in the flow of information.
- Decoder-only (e.g., GPT): Unidirectional. Information flows from left-to-right. The model's goal is to predict the next token.
- Encoder-only (e.g., BERT): Bidirectional. Information flows in all directions. The model's goal is to build the best possible representation for each token, using the full context of the sequence.
The diagram below contrasts these architectures, showing how BERT is architecturally just the encoder part of the original Transformer.
This architectural choice has profound implications, which are well-summarized in the following introductory text.
Encoder Architecture: Bidirectional Transformers for Understanding ...
This article from the "Language AI Handbook" clearly defines the encoder-only paradigm and outlines its advantages for understanding-based tasks.
Please read the introduction and the section titled The Encoder-Only Paradigm. Focus on the three key advantages it lists for this architecture: bidirectional context, computational efficiency, and simpler training.
The Heart of BERT: Bidirectional Self-Attention
The key mechanism that enables this deep understanding is bidirectional self-attention. Unlike the masked self-attention in a decoder, here there is no restriction on which tokens can be attended to. When computing the representation for a token at any position, the model can and does look at all tokens that come before it and all tokens that come after it.
Consider the sentence: "The bank approved the loan quickly."
To understand that "bank" refers to a financial institution and not a river bank, the word "loan" is a crucial piece of context. A decoder-only model processing "bank" would only have seen "The". An encoder-only model, however, sees the entire sentence, allowing "loan" to directly influence the final representation of "bank".
Mathematically, this is achieved by using the scaled dot-product attention formula without any masking:
The attention score matrix is a full matrix, not a lower-triangular one. Every token computes a relevance score with every other token.
The article "Encoder Architecture" provides an excellent mathematical breakdown and visualization of this process.
Encoder Architecture: Bidirectional Transformers for Understanding ...
This section dives into the mechanics of bidirectional self-attention, contrasting it with the causal attention you saw in the last lesson.
Read the section Bidirectional Self-Attention. Pay close attention to the visualizations comparing the full attention matrix of an encoder with the masked, lower-triangular matrix of a decoder. The Python code snippets also provide a clear implementation of the attention formula with and without the mask, which should resonate with your technical background.
The Training Dilemma and Its Solution
The power of bidirectionality introduces a training paradox. The standard language modeling objective of predicting the next token is no longer viable. If the model can see the entire sequence, predicting the next token is a trivial task of simply copying it.
To solve this, BERT's creators introduced two ingenious "pre-training" objectives that force the model to learn rich contextual representations bidirectionally. These tasks are not the final use case of the model but are designed to imbue it with a general understanding of language.
The video "BERT explained" by Umar Jamil provides a superb walkthrough of these concepts.
BERT explained: Training, Inference, BERT vs GPT/LLamA, Fine tuning, [CLS] token
This video clearly explains the conceptual differences between BERT and GPT-style models before diving into the specific training tasks that define BERT.
First, watch the segment that contrasts BERT with language models like GPT. This will frame the motivation for BERT's unique training objectives.
1. Masked Language Modeling (MLM)
Instead of predicting the next token, MLM is like a fill-in-the-blanks exercise. During pre-training, a certain percentage of the input tokens (15% in the original BERT) are randomly masked. The model's objective is to predict the original identity of these masked tokens, using the unmasked tokens on both sides as context.
To prevent a mismatch where the model sees [MASK] tokens during training but never during real-world use (fine-tuning), the masking strategy is nuanced:
- Of the 15% of tokens chosen for masking:
- 80% are replaced with the
[MASK]token. - 10% are replaced with a random token from the vocabulary.
- 10% are left unchanged.
- 80% are replaced with the
In all three cases, the model is trained to predict the original token at that position. This forces the model to maintain a rich representation for every token in the sequence.
BERT explained: Training, Inference, BERT vs GPT/LLamA, Fine tuning, [CLS] token
The video continues with a detailed explanation of the Masked Language Model task.
Watch the section explaining the Masked Language Model (MLM) task. The speaker details the 15% masking strategy and how the loss is calculated only at the masked positions.
2. Next Sentence Prediction (NSP)
Many important downstream tasks, such as Question Answering (QA) and Natural Language Inference (NLI), rely on understanding the relationship between two sentences. To equip BERT for this, it was also pre-trained on a Next Sentence Prediction task.
For this task, the model is given two sentences, A and B, and must predict whether B is the sentence that actually follows A in the original text corpus or if it's just a random sentence.
To handle this, the input is formatted in a specific way:
- A special
[CLS](classification) token is added to the start of the entire sequence. - The two sentences are separated by a
[SEP](separator) token. - Segment Embeddings are added to the token and positional embeddings to distinguish Sentence A from Sentence B.
The final hidden state corresponding to the [CLS] token is then used to make the binary prediction (IsNext vs. NotNext). This trains the [CLS] token to act as a meaningful aggregate representation of the entire input pair.
BERT explained: Training, Inference, BERT vs GPT/LLamA, Fine tuning, [CLS] token
Now, let's explore the second pre-training task and the special role of the [CLS] token.
Watch the section covering the Next Sentence Prediction (NSP) task. Pay attention to how the input is constructed with [CLS], [SEP], and segment embeddings. Then, watch the follow-up explanation on the function of the [CLS] token. The CEO analogy is a helpful way to understand its role as an information aggregator.
Fine-Tuning BERT for Downstream Tasks
After being pre-trained on MLM and NSP over a massive text corpus, BERT possesses a deep, bidirectional understanding of language. The real power of this pre-trained model is unlocked through fine-tuning.
Unlike generative models that are often used with prompting, BERT-style models are typically adapted to specific tasks by adding a small, task-specific neural network layer (a "head") on top of the pre-trained encoder. The entire model is then trained for a few epochs on a smaller, task-specific labeled dataset.
Here are a few common fine-tuning applications:
-
Text Classification (e.g., Sentiment Analysis): A single sentence is passed through BERT. The final output representation of the
[CLS]token is fed into a simple linear classifier to predict a label (e.g., "Positive", "Negative"). -
Sequence Labeling (e.g., Named Entity Recognition): A sentence is passed through BERT. The final output representation of each token is fed into a linear classifier to predict a tag for that token (e.g., B-PER, I-ORG, O).
-
Extractive Question Answering: The question and a context paragraph are concatenated and passed through BERT. Two separate heads are added to predict, for each token in the context paragraph, the probability of it being the start of the answer span and the probability of it being the end of the answer span.
The following video and article provide excellent conceptual demonstrations of this fine-tuning process.
BERT explained: Training, Inference, BERT vs GPT/LLamA, Fine tuning, [CLS] token
This final segment from the "BERT explained" video visualizes how the pre-trained BERT model is adapted for different understanding tasks.
Please watch the section on fine-tuning BERT. It clearly demonstrates the input/output flow for both text classification and question answering.
Encoder Architecture: Bidirectional Transformers for Understanding ...
This section of the "Encoder Architecture" article complements the video by showing how different fine-tuning heads operate on the encoder's output representations.
Read the section titled Encoder for Understanding Tasks. This will reinforce how the [CLS] token is used for classification and how token-level and span-level tasks are handled.
Conclusion
In this lesson, we have explored the encoder-only architecture, a powerful paradigm for natural language understanding. By leveraging bidirectional context, it builds rich representations that are highly effective for analytical tasks.
Here are the key takeaways:
- Architecture: An encoder-only model like BERT is a stack of Transformer encoder blocks. Its core mechanism is bidirectional self-attention, allowing every token to draw context from the entire input sequence.
- Suitability: This architecture is designed for understanding-based tasks (classification, extraction, entity recognition), not for text generation. Its single forward pass makes it very efficient for these tasks at inference time.
- Pre-training: Because standard autoregressive prediction is not possible, BERT is pre-trained on two novel objectives: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP).
- Adaptation: The pre-trained model is adapted to specific downstream applications via fine-tuning, where a small, task-specific classification head is added and trained on a labeled dataset.
You now understand the two primary branches of the Transformer family tree: the generative, unidirectional decoder-only models, and the understanding-focused, bidirectional encoder-only models. In our next and final lesson of this module, we will place all three architectures—encoder-decoder, decoder-only, and encoder-only—side by side to compare their objectives and applications, solidifying your high-level map of the Transformer landscape.