Hello! Welcome back to our course on Generative Sequence Modeling.
Introduction
In our last few lessons, we've built a Transformer block from the ground up, dissecting its key components: multi-head self-attention, feed-forward networks, residual connections, and LayerNorm. You now have a solid understanding of the fundamental building block of modern large language models.
Today, we move from the building block to the blueprint. We will explore how these blocks are arranged to create the three primary Transformer architectures. The learning outcome for this lesson is to distinguish encoder-decoder transformers from decoder-only transformers and identify their typical use cases. We will also cover the third variant, encoder-only models, to provide a complete picture.
This lesson is crucial because the choice of architecture fundamentally shapes a model's capabilities, determining whether it excels at understanding text, generating text, or transforming text from one form to another. Understanding these distinctions is essential as we move toward the advanced architectures mentioned in our syllabus, most of which are decoder-only models.
1. The Three Architectural Flavors of Transformers
At a high level, all Transformer-based models are assembled from the same core components, but they are arranged in one of three ways:
- Encoder-Only: Designed to build a rich numerical understanding of an input text.
- Decoder-Only: Designed to generate new text, token by token.
- Encoder-Decoder: Designed to transform an input sequence into a new output sequence.
To get a quick, high-level overview of these three families, please watch the following video.
Which transformer architecture is best? Encoder-only vs Encoder-decoder vs Decoder-only models
This video from the Efficient NLP channel provides a concise introduction to the three main Transformer architectures and their core purposes.
Watch the entire video (it's short). Focus on the distinct purpose of each architecture: encoder-only for understanding, decoder-only for generation, and encoder-decoder for sequence-to-sequence tasks.
The diagram below provides a helpful summary of the concepts discussed in the video, which we will be exploring in detail.
Now, let's dive into each of these architectures one by one.
2. Encoder-Decoder Models: The Original "Sequence-to-Sequence" Architecture
The first Transformer, introduced in the paper "Attention Is All You Need," was an encoder-decoder model designed for machine translation. This architecture is a natural fit for sequence-to-sequence (seq2seq) tasks, where an input sequence is transformed into an output sequence (e.g., translating English to French, summarizing a long article).
It consists of two stacks of Transformer blocks:
- The Encoder Stack: Its job is to process the entire input sequence and "encode" it into a set of rich contextual representations—one vector for each input token. The self-attention layers here are bidirectional, meaning every token can attend to every other token in the input.
- The Decoder Stack: Its job is to generate the output sequence autoregressively (one token at a time). It has two types of attention layers:
- Masked Self-Attention: This is the causal attention we've discussed before. It allows each newly generated token to attend to the previous tokens in the output sequence.
- Cross-Attention: This is the key mechanism that connects the encoder and decoder. In this layer, the queries () come from the decoder's masked self-attention sub-layer, while the keys () and values () come from the final output of the encoder. This allows the decoder, at every step of generation, to look back at the entire input sentence to decide which token to generate next.
The following video from HuggingFace provides an excellent step-by-step walkthrough of this process.
Transformer models: Encoder-Decoders
Let's watch a detailed explanation of how the encoder-decoder architecture works, using translation as a concrete example.
Watch from the timestamp 00:30 to 06:16. Pay close attention to: How the encoder's output is passed to the decoder (00:30-03:20). The role of cross-attention in the translation example (03:20-04:57). The discussion on why this architecture is suited for summarization and can handle different input/output lengths (04:57-06:16).
Use Cases:
Because the encoder can build a complete representation of the input and the decoder can generate a new, potentially different-length output based on that representation, this architecture excels at:
- Machine Translation: (e.g., T5, BART, Marian)
- Text Summarization
- Generative Question Answering
3. Encoder-Only Models: For Deep Understanding
What if your task doesn't require generating a new sequence, but rather understanding the input text to make a prediction? For example, classifying the sentiment of a sentence or identifying named entities (like people or places). For these tasks, the decoder part is unnecessary. This leads to encoder-only architectures.
- Architecture: A stack of encoder blocks.
- Attention: The self-attention is fully bidirectional. Each token can see and incorporate information from all other tokens in the sequence, both to its left and right. This allows the model to build a very deep contextual understanding.
- Pre-training: These models cannot be trained on simple next-token prediction. Instead, they are typically pre-trained using an objective called Masked Language Modeling (MLM). In MLM, some tokens in the input sentence are randomly hidden (masked), and the model is trained to predict what those original masked tokens were.
This process forces the model to learn the relationships between words in context.
Understanding Encoder And Decoder LLMs
The following article gives a clear explanation of encoder-only architectures and the Masked Language Modeling pre-training task.
Read the section titled 'Encoders'. Focus on the description of BERT and the illustration of masked language modeling. This will clarify how these models are trained for understanding rather than generation.
Use Cases:
Due to their deep bidirectional context, encoder-only models are best for analytical tasks:
- Text Classification: Sentiment analysis, topic classification.
- Named Entity Recognition (NER): Identifying entities like "Person," "Organization," "Location."
- Extractive Question Answering: Finding the answer to a question that is directly stated within a given text.
Examples: BERT, DistilBERT, RoBERTa.
4. Decoder-Only Models: The Modern Generative Workhorse
This brings us to the third and currently most prominent architecture: decoder-only models. The vast majority of modern LLMs, including the GPT series and the models featured in this course (Qwen, DeepSeek), are decoder-only.
- Architecture: A stack of decoder blocks.
- Attention: Self-attention is causal (or masked). Each token can only attend to itself and the tokens that came before it. This autoregressive property is what makes them natural text generators.
- Pre-training: They are trained on the simple and powerful objective of next-token prediction, which you are already familiar with.
They operate by taking an input sequence (a "prompt") and repeatedly generating the next token, feeding that new token back into the input for the next step.
The image below shows the structural difference, highlighting the absence of cross-attention in the decoder-only model.
To learn more about their capabilities and why they have become so dominant, let's turn to the Hugging Face LLM Course.
Transformer Architectures - Hugging Face LLM Course
This reading provides an excellent summary of decoder-only models and their evolution into modern LLMs.
Please read the sections 'Decoder models' and 'Modern Large Language Models (LLMs)'. The table under 'Key capabilities' is particularly useful for understanding the wide range of tasks these models can perform.
Use Cases:
Although they can be fine-tuned for a variety of tasks, decoder-only models are fundamentally text generators. They excel at:
- Open-ended text generation: Writing essays, stories, poems.
- Conversational AI / Chatbots (like ChatGPT).
- In-context learning: Performing tasks based on examples given in the prompt, without any weight updates.
- Code Generation.
- Tasks like summarization and translation, which can be framed as a text generation problem.
Examples: GPT series, LLaMA series, Gemma, and the models central to this course like DeepSeek-V3 and Qwen 2.5. The autonomous driving planners we will study, DAP and Plan-R1, are also built on decoder-only architectures.
Conclusion
In this lesson, we have mapped out the three fundamental architectures of Transformer models.
Key Takeaways:
| Feature | Encoder-Only (e.g., BERT) | Decoder-Only (e.g., GPT) | Encoder-Decoder (e.g., T5) |
|---|---|---|---|
| Blocks Used | Encoder stack only | Decoder stack only | Both Encoder and Decoder stacks |
| Attention | Bidirectional self-attention | Causal (masked) self-attention | Bidirectional self-attention in encoder; Masked self-attention and cross-attention in decoder |
| Primary Goal | Understand input text | Generate text | Transform input to output |
| Typical Tasks | Classification, NER | Open-ended generation, chat | Translation, summarization |
Understanding these architectural choices is the key to selecting the right tool for a job. For the remainder of this course, our focus will be primarily on decoder-only models, as they form the foundation of modern generative AI and the advanced planning systems we aim to deconstruct.
Preview of the Next Lesson:
Now that you understand what a decoder-only model is, we will look at exactly how it generates text. Our next lesson will cover the process of text generation via iterative next-token sampling, exploring different decoding strategies like greedy search and beam search.