Masked Language Modeling in BERT

The Limitation of Unidirectional Context

Causal Language Modeling is powerful for generation, but it has an inherent limitation for understanding tasks. Because it processes text strictly from left to right, it can never use "future" words to inform its understanding of a current word.

Consider this classic example:

1. I went to the bank to deposit money.
2. I sat by the river bank and watched the boats.

For a human, the context that follows the word "bank" ("deposit money" vs. "river") is crucial for disambiguation. A purely autoregressive model predicting the word "bank" would only see the preceding context (e.g., "I went to the..."). It lacks the full picture.

This is the core problem that Masked Language Modeling was designed to solve. It enables a model to look at a word's context from both the left and the right simultaneously.

The "Cloze" Task: Defining Masked Language Modeling

To force a model to learn from bidirectional context, the creators of BERT devised a new pre-training task. Instead of predicting the next word, the model is tasked with predicting a word that has been intentionally hidden, or "masked," within a sentence. This is analogous to a "Cloze test," a common reading comprehension exercise where you fill in the blanks in a passage.

For example, given the input sentence:
The [MASK] sat on the mat.

The model's job is to predict that the [MASK] token should be "cat". To do this successfully, it must use the context from both sides: "The..." and "...sat on the mat."

This "hide and predict" objective is the essence of Masked Language Modeling. The video below introduces this idea as the motivation for BERT and contrasts it with the causal modeling used in GPT.

As the video explained, this objective is a natural fit for the Transformer encoder architecture. Unlike the decoder, which uses causal masking to restrict information flow, the encoder's self-attention mechanism is fully connected. Each token's representation is built by attending to every other token in the input sequence, making it inherently bidirectional.

The Training Procedure: The 80-10-10 Masking Strategy

The core idea of "hide and predict" is simple, but a naive implementation would create a problem. If we always replace the words we want to predict with a special [MASK] token, the model will be trained on inputs containing these [MASK] tokens but will never see them during real-world use (fine-tuning or inference). This mismatch between the pre-training and downstream tasks could degrade performance.

To mitigate this, BERT's authors introduced a more nuanced strategy. During pre-training, they randomly select 15% of the tokens in each sequence for prediction. For these selected tokens, they apply the following "80-10-10" rule:

• 80% of the time: The token is replaced with the [MASK] token.
  • the cat sat -> the [MASK] sat
• 10% of the time: The token is replaced with another random token from the vocabulary.
  • the cat sat -> the dog sat
• 10% of the time: The token is left unchanged.
  • the cat sat -> the cat sat

In all three cases, the model's objective remains the same: predict the original word at that position (in this case, "cat").

This strategy is clever for a few reasons:

1. The 80% [MASK] replacement primarily drives the learning of context.
2. The 10% random replacement forces the model to learn to correct a corrupted input, making it more robust. It can't just trust that every word it sees is correct.
3. The 10% unchanged replacement is the most subtle. It forces the model to learn a rich representation for every word, not just copy its input. Since it might be asked to predict a word that is already present, it must rely on the surrounding context to confirm the word's identity. This helps to bridge the pre-train/fine-tune discrepancy.

The following reading provides a detailed breakdown of these concepts.

Bidirectional Attention in Practice

We've established that MLM relies on bidirectional context, and that the encoder architecture is inherently bidirectional. But how does this work at the level of the attention mechanism?

Recall from our previous module that causal masking in a decoder is achieved by modifying the attention score matrix $Q{K}^T$, setting all values above the main diagonal to negative infinity before the softmax step. This prevents a token from attending to any subsequent tokens.

For a bidirectional encoder used in MLM, the solution is simple: we don't apply this mask. The self-attention mechanism is allowed to operate in its natural state, where every query vector can attend to every key vector in the sequence. This allows information to flow freely in both directions, enabling the model to build a deeply contextualized representation for each token before predicting the masked ones.

This final video segment provides a clear, technical explanation of this process.

Conclusion

In this lesson, we have dissected Masked Language Modeling, the pre-training objective that underpins encoder models like BERT. It represents a fundamental departure from the autoregressive approach of generative models.

Here are the key takeaways:

• Definition: MLM is a pre-training objective where a model learns to predict tokens that have been randomly masked from the input. This is often called a "Cloze" or "fill-in-the-blanks" task.
• Bidirectional Context: By predicting a word from its full surrounding context (both left and right), the model learns deep, bidirectional representations of language. This is crucial for tasks requiring nuanced understanding, such as sentiment analysis, named entity recognition, and question answering.
• Architectural Alignment: MLM is the natural objective for encoder-only Transformer architectures, which use an unmasked self-attention mechanism that allows every token to attend to every other token in the sequence.
• Training Strategy: The 80-10-10 masking strategy is used to make the model more robust and to bridge the gap between the pre-training objective (which involves [MASK] tokens) and downstream tasks (which do not).

You now understand the two primary pre-training objectives for Transformers: Causal Language Modeling for generation and Masked Language Modeling for understanding.

In our next lesson, we will zoom in on a component common to both approaches: the final output layer. We will explore how both generative and masked models take their internal vector representations and transform them into a probability distribution over the entire vocabulary to make their final token predictions.