Welcome back. In our previous lesson, we established a firm statistical and information-theoretic foundation for the cross-entropy loss function. You learned that minimizing this loss is equivalent to maximizing the likelihood of the training data, and that for next-token prediction, it elegantly simplifies to minimizing the negative log probability of the correct token, .
This leaves us with a critical operational question. To calculate this loss, the model needs to compare its prediction to the "correct" next token. But in a long sequence, how does the model get the correct context for each prediction it makes during training? This lesson will answer that question by dissecting the fundamental, and surprisingly different, mechanisms used for training versus inference in autoregressive models.
Your goal is to understand teacher forcing, the highly efficient method used for training, and contrast it with autoregressive generation, the sequential process used for inference. We will explore not just how they work, but why this duality exists and what trade-offs it entails.
The Two Modes of a Transformer: Training vs. Inference
A generative Transformer operates in two distinct modes. At inference time, its job is to generate new text from scratch. It has no ground truth to guide it. In contrast, at training time, its goal is to learn from a vast dataset of correct examples. These two different goals lead to two very different data flow strategies.
Inference: Autoregressive Generation
Let's start with inference, as it is perhaps more intuitive. When we ask a model to generate text, it does so autoregressively, meaning it generates one token at a time, and each new token depends on the sequence of tokens it has generated so far.
Imagine writing a story one word at a time. The next word you choose depends on the sentence you've written up to that point. The model does exactly this:
- It starts with an initial context (e.g., a prompt and a
[START]token). - It performs a forward pass and predicts the very first token.
- This newly generated token is appended to the input sequence.
- The model then takes this new, longer sequence as its input for the next step.
- This loop continues, token by token, until the model generates an
[END]token or reaches a predefined length limit.
This process is inherently sequential and computationally intensive, as each step requires a full forward pass through the decoder.
The following video provides an exceptionally detailed, step-by-step walkthrough of this autoregressive inference process. Watching it will give you a concrete mental model of the data flow.
How a Transformer works at inference vs training time
This video by Niels Rogge, a research scientist at Hugging Face, meticulously illustrates how an encoder-decoder Transformer like T5 works at inference time. While the example is a translation model, the core autoregressive decoder logic applies directly to decoder-only models like GPT as well.
Watch the segment from the beginning of the inference explanation up to the transition to training time (Inference Time Walkthrough). As you watch, focus on these key stages: One-time Encoding: The source sentence is tokenized and fed through the encoder only once. The resulting hidden states are cached. The Generation Loop: Notice how the process is broken into discrete time steps. Decoder Input: At each time step t, the input to the decoder consists of all tokens generated up to time step t-1. Prediction: The model predicts the next token by taking the argmax (in the case of greedy decoding) of the logits for the final token in the sequence. Appending the Output: The predicted token ID is added to the decoder's input for the next time step.
Training: Teacher Forcing
If we used the slow, sequential autoregressive method for training, it would be prohibitively inefficient. Furthermore, early in training, the model's own predictions are mostly noise. Forcing the model to learn from its own mistakes would be like asking a student driver to learn by navigating a city with a perpetually incorrect map—it's unstable and leads nowhere fast.
The solution is a technique called teacher forcing. During training, we have access to the complete, correct target sequence (the "ground truth"). Instead of feeding the model its own, potentially flawed, predictions, we always feed it the correct token from the training data as input for the next step.
The "teacher" (the ground-truth data) "forces" the model to stay on the correct path, providing a stable and accurate context at every step. This makes the optimization process much more stable and efficient.
The Power of Parallelization
Teacher forcing's true power lies in the computational efficiency it unlocks. Since the entire input sequence for the decoder is known ahead of time (it's simply the ground-truth target sequence, shifted), we don't need to generate tokens one by one. We can feed the whole sequence to the decoder and compute the predictions for every single position in parallel within a single forward pass.
How does the model avoid "cheating" by looking at future tokens in the sequence? This is where the causal attention mask, which we've discussed previously, plays its crucial role. During this single, parallel forward pass, the mask ensures that the prediction for token at position is computed using attention only over tokens from positions to . It mathematically enforces the autoregressive property while allowing for massive computational parallelism.
Let's return to the video from Niels Rogge, which now explains the training process.
How a Transformer works at inference vs training time
Now, let's watch the second part of the same video, which covers the training process and explicitly details teacher forcing.
Watch the segment on training time (Training Time Walkthrough). Pay close attention to: A Single Forward Pass: Notice how the entire process is described as a single, non-iterative operation. labels and decoder_input_ids: The key insight is here. The labels are the ground-truth target sequence. The decoder_input_ids are created automatically by shifting the labels one position to the right and prepending the [START] token. This is teacher forcing in action. Loss Calculation: The cross-entropy loss is computed for all token positions simultaneously by comparing the model's parallel predictions (logits) against the labels. Causal Masking: The video mentions the causal attention mask (this section) as the mechanism that prevents the decoder from looking into the future, even though the full sequence is provided as input.
To crystallize this central idea of parallelization, watch this very short, incisive clip.
Here is how Transformers ended the tradition of Inductive Bias in Neural Nets
This clip from the channel Neural Breakdown with AVB directly addresses the efficiency difference between training and inference.
Watch the segment titled Why Training is Parallel. It perfectly summarizes why the availability of the target sequence during training is the key that unlocks parallel processing, a feat impossible during inference.
The Trade-off: Exposure Bias
While teacher forcing is essential for efficient and stable training, it introduces a subtle but important discrepancy between the training and inference environments. This is known as the train-inference mismatch or exposure bias.
- During training, the model is only ever exposed to perfectly correct sequences provided by the "teacher." It never learns how to recover from its own mistakes.
- During inference, the model is on its own. It must generate sequences based on its own previous outputs, which may contain errors. A single bad token can lead the model down a path of compounding errors, derailing the entire generation.
Despite this theoretical problem, teacher forcing remains the standard for training state-of-the-art language models. The negative effects of exposure bias are largely mitigated by using massive, diverse training datasets and sophisticated decoding strategies (like beam search or nucleus sampling, which we will cover in a later module) that help guide the model during inference.
This article provides a concise and clear definition of teacher forcing and exposure bias.
Transformer Training Essentials | CodeSignal Learn
This section from a CodeSignal Learn article formally defines teacher forcing and its primary drawback.
Please read the section titled Teacher Forcing: Training vs Inference. Focus on how it clearly articulates the difference between the context seen during training (perfect tokens) and inference (model's own predictions), and names the resulting issue "train-inference mismatch" or "exposure bias".
Conclusion
You now understand the critical distinction between how a Transformer operates during training versus inference. This duality is a clever engineering solution that balances the need for efficient, stable learning with the reality of sequential, open-ended generation.
Let's summarize the key takeaways in a table:
| Feature | Training (with Teacher Forcing) | Inference (Autoregressive Generation) |
|---|---|---|
| Decoder Input | Ground-truth tokens from the dataset (shifted right) | The model's own previously generated tokens |
| Computation | A single, parallel forward pass for the entire sequence | An iterative loop of sequential forward passes, one per token |
| Speed | Fast, highly efficient, and scalable | Slow and inherently sequential |
| Key Enabler | Causal Attention Mask (prevents cheating) | The generation loop itself |
| Primary Drawback | Exposure Bias: Model never sees its own mistakes. | Error Propagation: Mistakes can compound and derail generation. |
We have now completed our survey of the foundational training objectives and mechanics of a single model. We know what the model is optimizing (cross-entropy loss) and how the data flows during that optimization (teacher forcing). In our next module, we will zoom out from the single training step to explore the enormous engineering effort required to execute this process at scale, covering topics like optimizers, learning rate schedules, and the distributed training techniques that make today's massive LLMs possible.