Introduction
Welcome back. In our previous lessons, we've drilled down into the heart of the Transformer: the attention mechanism. You've seen how it works, what makes it computationally expensive (O(n²)), and how brilliant optimizations like FlashAttention and sparse attention have made it possible to handle increasingly long sequences. These mechanisms are the fundamental building blocks of modern LLMs.
Today, we zoom out from these individual blocks to see the blueprints of the buildings they create. The learning outcome for this lesson is to compare decoder-only (e.g., GPT), encoder-only (e.g., BERT), and encoder-decoder (e.g., T5) architectures.
Understanding these three architectural families is one of the most critical high-level concepts in the field. As an applied AI engineer, knowing which architecture is suited for a particular task—be it analysis, generation, or transformation—is fundamental to effective model selection and system design. We will dissect each architecture, understand its core function, and map it to its ideal use cases.
The Three Architectural Families
At their core, the three major Transformer architectures are designed for distinct purposes: understanding, generating, and transforming text.
- Encoder-Only Models: These are designed for understanding. They read an entire piece of text and produce a rich numerical representation of it. Think of them as comprehension engines.
- Decoder-Only Models: These are designed for generating. They take a prompt and continue writing, producing new text token by token. Think of them as creative writers.
- Encoder-Decoder Models: These are designed for transforming one sequence into another. They first read an input sequence to understand it, and then generate a new output sequence based on that understanding. Think of them as translators.
The key difference between them lies in how they use the attention mechanism you've already studied. Let's start with a video that provides a concise overview of all three.
Which transformer architecture is best? Encoder-only vs Encoder-decoder vs Decoder-only models
This video from the Efficient NLP channel provides an excellent 7-minute summary of the three architectures, their functions, and their typical use cases. It will give you a strong mental model before we dive into each one.
Watch the entire video (from 00:00 to 07:18). Pay attention to the primary function described for each architecture: producing embeddings (encoder), generating text (decoder), and sequence-to-sequence tasks (encoder-decoder).
Now, let's solidify these concepts using a clear visual and a more detailed textual resource.
Transformer Architectures - LLM Course
The Hugging Face LLM course provides a well-written guide to these architectures. We will use it as our primary reference for this lesson.
First, read the introductory section of the document, up to (but not including) the 'Encoder models' heading. This will set the stage for our deeper dive.
1. Encoder-Only Models: Masters of Comprehension
Encoder-only models, famously represented by BERT (Bidirectional Encoder Representations from Transformers), are architecturally the simplest. They consist of a stack of encoder blocks.
Core Idea: Bidirectional Context
The defining feature of an encoder is its bidirectional self-attention. When processing an input sentence, every token's representation is built by attending to every other token in the sentence, both before and after it. This allows the model to build a deep, contextual understanding of the entire input at once. Because they are trained to reconstruct corrupted inputs, they are also known as auto-encoding models.
Use Cases:
This deep understanding makes encoder models ideal for Natural Language Understanding (NLU) tasks, where the goal is to analyze or classify the input text, not generate new text.
- Text Classification: Sentiment analysis, topic categorization, spam detection.
- Named Entity Recognition (NER): Identifying people, places, and organizations in text.
- Extractive Question Answering: Finding the answer to a question within a given passage of text.
Transformer Architectures - LLM Course
Let's read the relevant section from the Hugging Face course to detail the encoder architecture.
Read the section titled 'Encoder models'. Note the emphasis on 'bi-directional' attention and its suitability for tasks requiring a full-sentence understanding.
For additional depth, the "Datafuse Analytics" video provides a historical walkthrough of the evolution of important encoder-only models. This is optional but highly recommended if you're curious about the lineage of models like BERT, RoBERTa, and DeBERTa.
Confused which Transformer Architecture to use? BERT, GPT-3, T5, Chat GPT? Encoder Decoder Explained
This optional video explores the family tree of encoder-only models, which can provide valuable context on how they have evolved.
If you have time, watch the segment from 01:24 to 08:20. It covers the pre-training objectives and improvements of several key encoder models.
2. Decoder-Only Models: Fluent Generators
Decoder-only models, with the GPT (Generative Pre-trained Transformer) family as the primary example, have become the dominant architecture for large language models today.
Core Idea: Autoregressive Generation
The key mechanism here is masked self-attention, also called causal attention. When generating the next token, the model can only attend to the tokens that came before it. It cannot see "into the future." This process of predicting the next token based on the previous ones is called autoregression. These are often called auto-regressive models.
Use Cases:
This architecture is inherently suited for Natural Language Generation (NLG) tasks.
- Open-ended Text Generation: Writing articles, stories, and poetry.
- Conversational AI: Powering chatbots and virtual assistants like ChatGPT.
- In-context learning: Solving tasks with few-shot prompting, where examples are provided directly in the input.
The primary way to control these models is through prompting, where the input text provides the context and instruction for the desired output. Mastering this interaction is a core skill for an LLM engineer.
Transformer Architectures - LLM Course
Now, let's explore the decoder architecture in the Hugging Face course.
Read the section 'Decoder models'. Pay close attention to the term 'auto-regressive' and the table of capabilities, which highlights their versatility.
Test your understanding!
Why is bidirectional attention unsuitable for a generative task like writing a story, and why is causal attention perfect for it?
Show answer
Bidirectional attention would be unsuitable because for the model to predict the next word, it would be "cheating" if it could already see all the words that follow it. There would be no learning challenge.
Causal (masked) attention is perfect because it mimics the natural process of generation. To decide on the next word, you only have the context of the words you've already written. This forces the model to learn the patterns, structure, and semantics of language to make a coherent prediction.
3. Encoder-Decoder Models: Expert Transformers
Encoder-decoder models, also known as sequence-to-sequence (seq2seq) models, were the architecture of the original Transformer paper. Models like T5 and BART are modern examples.
Core Idea: Conditioned Generation via Cross-Attention
This architecture combines the strengths of the other two:
- The Encoder reads the entire source sequence using bidirectional self-attention, creating a rich representation of its meaning.
- The Decoder then generates the target sequence autoregressively (using causal self-attention on its own outputs), but with a crucial addition: cross-attention. At each step, the decoder not only looks at the tokens it has already generated but also attends to the encoder's final output.
This cross-attention mechanism allows the decoder to be constantly conditioned on the full meaning of the input sequence while it generates the output.
Use Cases:
This architecture excels at tasks that require transforming an input sequence into a new output sequence.
- Machine Translation: Translating a sentence from English to French.
- Summarization: Condensing a long article into a short paragraph.
- Generative Question Answering: Answering a question based on a provided context, where the answer is a newly generated sentence, not just an extracted phrase.
Transformer Architectures - LLM Course
Finally, let's cover the sequence-to-sequence architecture.
Read the section 'Sequence-to-sequence models'. Focus on understanding the role of the encoder (understanding input) and the decoder (generating output), and how they are linked.
Choosing the Right Architecture
Your ability to map a business problem to the correct model architecture is a valuable engineering skill. The choice almost always comes down to the nature of the task.
The article "Mastering Language Model Architectures" provides simple Python code examples using the transformers library pipeline, which can help make these concepts more concrete.
Mastering Language Model Architectures: A ...
This article offers practical code snippets that demonstrate how to use each type of model for its canonical task.
Skim through sections 1, 2, and 3 ('Encoder-Only Models', 'Decoder-Only Models', and 'Encoder-Decoder Models'). You don't need to run the code, but observe how each pipeline call maps directly to a specific task: sentiment analysis (encoder), text generation (decoder), and summarization (encoder-decoder).
To summarize the decision process, refer to this handy table from the Hugging Face course.
Transformer Architectures - LLM Course
This final section provides a clear, task-oriented guide for model selection.
Read the final section, 'Choosing the right architecture'. The table is an excellent summary and a great reference for future work.
Final Challenge: Apply Your Knowledge
For each of the following scenarios, identify the best-fit Transformer architecture and justify your choice in one sentence.
- A system to automatically extract key financial figures (e.g., revenue, net income) from quarterly earnings reports.
- An internal tool that generates Python docstrings for functions based on the function's code.
- A feature for a news app that provides a one-paragraph summary of a full news article.
Show answer
- Encoder-Only (e.g., BERT): This is a Named Entity Recognition / extractive task that requires understanding the full context of the document to identify and pull out specific pieces of information.
- Decoder-Only (e.g., GPT): This is a generative task (code-to-text) where the model needs to produce fluent, descriptive text based on the provided code context.
- Encoder-Decoder (e.g., T5/BART): This is a classic sequence-to-sequence summarization task, where the model must first comprehend the entire article (encoder) and then generate a new, shorter sequence that captures the main points (decoder).
Conclusion
In this lesson, you've moved from the component level (attention) to the system level (architecture). You can now differentiate between the three major families of Transformer models and, more importantly, can reason about which architecture is the right tool for a given job.
Key Takeaways:
- Encoder-Only (e.g., BERT): Built for understanding. Uses bidirectional attention to create rich representations. Ideal for NLU tasks like classification and extraction.
- Decoder-Only (e.g., GPT): Built for generation. Uses causal (masked) attention for autoregressive text production. Dominant in modern LLMs for creative and conversational tasks.
- Encoder-Decoder (e.g., T5): Built for transformation. Combines a bidirectional encoder and a cross-attending decoder. Perfect for seq2seq tasks like translation and summarization.
What's Next?
You have now completed the foundational modules on sequence modeling and the core Transformer architecture. In the next lesson, we will begin Module 4: Modern LLM Architectures and Optimizations. We'll kick it off by exploring the Mixture-of-Experts (MoE) architecture. This is a cutting-edge technique used in models like Mixtral and GPT-4 to dramatically increase model size (and capability) while keeping computational costs manageable—a key innovation in the evolution of the decoder-only models we discussed today.