Introduction to Autoregressive Language Modeling

Introduction

In our last two lessons, we established the foundational pipeline for processing text:

1. Tokenization: We learned how to convert a raw string of text into a sequence of integer token IDs (e.g., [11241, 4712, 318, 12691, 0]).
2. Embedding: We then saw how an embedding layer maps these discrete IDs into dense, meaningful vectors, resulting in a tensor of shape (batch_size, sequence_length, embedding_dim).

Now we have a numerical representation of text that a neural network can process. The fundamental question is: what is the core task the model is trained to perform with these vectors?

This lesson answers that question by introducing autoregressive language modeling. You will learn that the primary objective of a decoder-only model like GPT is to predict the next token in a sequence given all the tokens that came before it. This simple yet powerful principle is the engine behind the generative capabilities of all modern large language models.

The Core Idea: A Next-Token Predictor

At its heart, a large language model is a sophisticated function that predicts what word (or token) comes next. The model generates long, coherent responses not by planning the whole text at once, but by performing this next-token prediction task iteratively.

This short, high-level video from 3Blue1Brown provides an excellent intuition for this process.

The process described in the video—where the model's output at one step becomes part of the input for the next—is known as autoregression. The term "auto" signifies that the model relies on its own past outputs to generate future ones.

This image provides a clear, step-by-step visualization of the autoregressive process:

Formalizing Autoregressive Language Modeling

We can formalize this intuitive idea using the language of probability. The goal of a language model is to learn the probability distribution over sequences of tokens, which we can write as $p(⟨w_i{⟩}_{i=1}^L)$, where $⟨w_i⟩$ is a sequence of $L$ tokens.

Using the chain rule of probability, we can decompose this joint probability into a product of conditional probabilities:
$p(w_1,w_2,\dots ,w_L)=p(w_1)\cdot p(w_2|w_1)\cdot p(w_3|w_1,w_2)\cdots p(w_L|w_1,\dots ,w_{L-1})$
This can be written more compactly as:
$p(⟨w_i{⟩}_{i=1}^L)={\prod }_{i=1}^{L}p(w_i|⟨w_j{⟩}_{j=1}^{i-1})$

This equation is the mathematical definition of autoregressive language modeling. It states that the probability of an entire sequence is the product of the probabilities of each token given all the tokens that came before it. The core task for the neural network is to learn how to effectively model this conditional probability, $p(\text{next token}|\text{previous tokens})$.

This resource provides a clear mathematical definition and explains how it leads to the generative process.

Because the prediction for any token $w_i$ only depends on "past" tokens ($w_1,\dots ,w_{i-1}$), this is also referred to as causal language modeling. The model cannot look ahead in the sequence during training.

A Concrete Example: The Bigram Model

To make this concept tangible, let's build the simplest possible character-level language model: a bigram model. This model makes a strong simplifying assumption: the probability of the next character depends only on the single most recent character.
$p(w_i|w_1,\dots ,w_{i-1})\approx p(w_i|w_{i-1})$
While this is a very basic model, implementing it from scratch is an excellent way to understand the mechanics of autoregressive generation. Andrej Karpathy's "makemore" tutorial does this brilliantly.

The following segments from his video will walk you through:

1. How to estimate the probabilities $p(w_i|w_{i-1})$ by simply counting character pairs in a large dataset.
2. How to use these probabilities to autoregressively sample new sequences of characters.

As you saw, even this simple bigram model can generate name-like sequences, though they are often nonsensical. The poor quality highlights a key point: the model's context is too small. It only knows about the single preceding character. This limitation is precisely what more complex architectures like the Transformer are designed to overcome, by allowing the model to consider a much longer history of previous tokens when making its next prediction.

Tying It All Together

We can now summarize the entire process for a causal language model. This diagram illustrates the key stages.

The key steps in Autoregressive Inference are:

1. Start with a prompt (e.g., "The quick brown fox").
2. Pass the sequence of tokens through the model to get a probability distribution for the next token.
3. Sample a token from this distribution (e.g., "jumps").
4. Append the new token to the sequence ("The quick brown fox jumps").
5. Repeat from step 2 until an end-of-sequence token is generated or a desired length is reached.

This article provides a concise written summary of this autoregressive loop.

Conclusion

In this lesson, you learned about the fundamental operating principle of generative language models.

Key Takeaways:

• Autoregressive Language Modeling is the task of predicting the next token in a sequence given all the tokens that came before it.
• This task is formalized by modeling the conditional probability distribution: $p({\text{token}}_i|\text{previous tokens})$.
• Text generation is an iterative (autoregressive) process: the model predicts one token, appends it to its input, and repeats the process to generate a sequence.
• The quality and coherence of the generated text depend heavily on the amount of prior context the model can consider. A simple bigram model has very limited context, while a Transformer can attend to the entire history.

Preview of the Next Lesson:

We've established that the model's job is to produce a probability distribution for the next token. But how does a neural network, which outputs raw numerical values (called logits), produce a valid probability distribution? And how do we measure how "good" or "bad" its prediction is during training?

In the next lesson, we will answer these questions by exploring two essential components:

1. The Softmax function, which converts logits into probabilities.
2. The Cross-Entropy Loss, which quantifies the error in the model's prediction, driving the learning process.