Comparing Transformer Architectures: Training and Applications

The Three Flavors of Transformer Architecture

As you now know, the original Transformer introduced in "Attention Is All You Need" consisted of both an encoder and a decoder. However, subsequent research found that using just one of these components could create powerful, specialized models. This led to the three dominant architectural paradigms we see today.

To start, let's get a high-level overview of these three model types and their primary roles. The following video provides an excellent introduction.

As the video explains, the core differences boil down to how information flows through the model and what the model is designed to do: understand, generate, or transform.

Let's visualize this architectural split. The key distinction lies in the type of attention mechanisms used and how the encoder and decoder components (or lack thereof) are connected.

This diagram neatly summarizes the structural differences:

• Encoder-Only (e.g., BERT): Processes the entire input sequence at once using bidirectional self-attention. It excels at creating a deep understanding of the input.
• Decoder-Only (e.g., GPT): Processes the input sequence token-by-token from left to right using masked (causal) self-attention. It is designed for generating new text that continues from a prompt.
• Encoder-Decoder (e.g., T5, BART, original Transformer): A hybrid model. The encoder processes the input sequence bidirectionally, and the decoder generates an output sequence autoregressively. Critically, the decoder uses cross-attention to look at the encoder's output, making its generation conditional on the entire input.

Training Objectives: Shaping Model Behavior

A model's architecture dictates its capabilities, but its pre-training objective is what truly teaches it how to behave. The task a model is trained on fundamentally shapes its understanding and use of language.

The following reading from a detailed book chapter on Transformers provides a clear narrative connecting architecture to pre-training and use cases.

Let's formalize the pre-training objectives:

1. Encoder-Only (BERT-style): These models cannot be trained with simple next-token prediction due to their bidirectional nature. Instead, they use "self-supervised" objectives that create a task from the input data itself.

   • Masked Language Modeling (MLM): As you learned, this involves randomly masking ~15% of the input tokens and training the model to predict the original tokens based on the surrounding bidirectional context. The objective is to minimize the loss on predicting these masked tokens.
   • Next Sentence Prediction (NSP): A binary classification task to determine if two sentences are sequential. This helps the model learn sentence-level relationships.

2. Decoder-Only (GPT-style): These are trained on the classic Causal Language Modeling (CLM) objective, also known as autoregressive language modeling.

   • Given a sequence of tokens $x_1,...,x_{t-1}$, the model is trained to predict the next token $x_t$. The objective is to maximize the likelihood of the entire training corpus, which factorizes into a product of conditional probabilities:
     $P(X)={\prod }_{t=1}^{T}P(x_t|x_1,...,x_{t-1})$
   • This naturally trains the model to generate coherent continuations of text.

3. Encoder-Decoder (T5/BART-style): These models are trained on sequence-to-sequence tasks. Modern variants use sophisticated denoising objectives.

   • T5 (Text-to-Text Transfer Transformer): T5 frames every NLP task as a text-to-text problem. Its pre-training objective involves corrupting the input by replacing random spans of text with a special sentinel token and training the model to output the missing text spans.
   • BART (Denoising Autoencoder for Sequence-to-Sequence): BART corrupts the input text in various ways (e.g., token masking, token deletion, sentence permutation) and trains the model to reconstruct the original, clean text. This forces the encoder to learn a robust representation and the decoder to generate language based on it.

Applications and Performance Trade-offs

The combination of architecture and pre-training objective directly leads to the strengths and weaknesses of each model family. The following table provides a high-level summary.

This table is a great reference, but let's add some empirical weight to these claims. A formal comparative analysis provides valuable insights into how these models perform on standardized benchmarks. The following paper analyzes model performance on tasks designed to test understanding versus those that test generalization.

The data from the paper confirms our theoretical understanding:

• For Natural Language Understanding (NLU): Tasks like sentiment analysis, textual entailment, and extractive question answering require a deep, bidirectional understanding of context. The SuperGLUE benchmark results show that encoder-only models (like DeBERTa, ERNIE) and encoder-decoder models (like T5) are superior. They achieve high scores with greater parameter efficiency than decoder-only models on these tasks.

• For Generalization and Generation (NLG): Tasks that require generating fluent text or applying knowledge to new problems with little to no specific training (few-shot learning) are the domain of decoder-only models. The MMLU benchmark results show that decoder-only models (GPT series, PaLM) overwhelmingly dominate. Their autoregressive pre-training makes them exceptionally good at in-context learning and creative generation.

This leads to a clear separation of typical applications:

Conclusion

In this final lesson of the module, we have synthesized our knowledge of the three foundational Transformer architectures. By comparing them side-by-side, a clear picture of their design philosophies and practical trade-offs emerges.

Here are the key takeaways:

• Three Families: The Transformer world is primarily divided into encoder-only, decoder-only, and encoder-decoder models.
• Attention is Key: The type of attention—bidirectional self-attention, masked self-attention, and cross-attention—is the defining architectural feature that enables or restricts information flow.
• Training Dictates Function: The pre-training objective (Masked LM, Causal LM, or Denoising Seq2Seq) directly shapes what the model learns and excels at.
• No Single Best Model: Encoder-only models are more efficient and powerful for NLU tasks. Decoder-only models are unmatched in generative capabilities and few-shot learning. Encoder-decoder models provide a robust framework for tasks that require transforming an input sequence into a new output sequence.

You now have a solid high-level understanding of the main Transformer architectures. This provides the context for our next module, where we will dive deeper into the mathematics and mechanics of Language Modeling and Training Objectives. We've discussed what these objectives are; next, we'll explore how they are implemented with loss functions like cross-entropy and training techniques like teacher forcing.