The LLM Training Pipeline: Pre-training to RLHF

The Three-Stage Rocket: An Overview

Creating a model like ChatGPT is not a single training event but a carefully orchestrated sequence of steps, each with a distinct purpose. Think of it as building and launching a three-stage rocket. Each stage builds upon the last, progressively refining the model's capabilities.

As you can see, the process starts with a massive, general-purpose phase and becomes increasingly specialized and data-curated. Let's examine each stage in detail.

Stage 1: Pre-training — Building a Universe of Knowledge

The first stage is by far the most computationally expensive. Its goal is to imbue the model with a broad understanding of language, grammar, facts, reasoning abilities, and world knowledge.

At its core, the pre-training objective is surprisingly simple: next-word prediction. The model is fed trillions of tokens from a massive, diverse corpus of text—essentially, a huge chunk of the public internet, books, and other sources—and its only job is to predict the next token in the sequence. By doing this over and over, it learns the statistical patterns of language.

Andrej Karpathy, a leading AI researcher, offers a powerful analogy for this process.

The output of this stage is a base model. This model is a powerful text-completion engine but is not yet a helpful assistant. It has no concept of following instructions; if you give it a question, it might just give you more questions that are stylistically similar to what it saw online. It is, as the article below describes, an "untamed monster."

Stage 2: Supervised Fine-Tuning (SFT) — Teaching the Model to Converse

The base model knows a lot, but it doesn't know how to apply that knowledge in a helpful, conversational way. The second stage, Supervised Fine-Tuning (SFT), is designed to teach it just that.

In SFT, the training data shifts dramatically from quantity to quality. Instead of the messy, unfiltered internet, we use a much smaller, curated dataset of high-quality prompt-and-response pairs. These pairs are often written by human labelers and demonstrate the desired behavior—how a helpful assistant should respond to a specific instruction or question.

The model is then fine-tuned on this dataset using standard supervised learning. The goal is to adjust the model's parameters so that when given a prompt from the dataset, its output looks as close as possible to the human-written ideal response.

After SFT, we have an assistant model. It's much more useful and aligned with user intent. However, creating these high-quality datasets is expensive and time-consuming, so they are relatively small. This can lead to the model overfitting and not generalizing well to prompts it hasn't seen. Furthermore, for many prompts, "goodness" is subjective and hard to capture in a single "correct" answer. This is where the final stage comes in.

Stage 3: Reinforcement Learning from Human Feedback (RLHF) — Refining with Preferences

RLHF is the step that gives modern chatbots their nuance and impressive ability to align with complex human values. The key insight behind RLHF is that it's often easier for people to compare and rank several responses than it is to write a perfect one from scratch.

The RLHF process has two main parts:

1. Training a Reward Model (RM): First, we take the SFT model and generate several different responses to a variety of prompts. We then show these responses to human labelers and ask them to rank them from best to worst. This comparison data—(prompt, winning_response, losing_response)—is used to train a separate model, the Reward Model. The RM's job is to act as a proxy for human judgment. It learns to take a prompt and a response and output a scalar score (a "reward") that predicts how much a human would like that response.

2. Fine-tuning the LLM with Reinforcement Learning: The SFT model is then further fine-tuned using a reinforcement learning algorithm, typically Proximal Policy Optimization (PPO). In this loop:

   • The LLM (the "policy") generates a response to a prompt.
   • The Reward Model scores that response, providing a reward.
   • The PPO algorithm updates the LLM's weights to make it more likely to produce responses that will receive a high reward.

A crucial element here is a KL divergence penalty. Given your computer science background, you can think of this as a regularization term in the objective function. It penalizes the LLM for deviating too far from the response style it learned during SFT. This prevents the model from "reward hacking"—discovering strange, nonsensical outputs that trick the reward model into giving high scores—and ensures the model's outputs remain coherent.

The StatQuest channel provides an exceptionally clear visual explanation of this entire process.

To round out your understanding with a more detailed text-based explanation, let's return to the tutorial article.

Your Project: "The AI Apprentice" Quest

Now, let's connect this back to your goal of building a game-based GenAI course. Explaining this three-stage pipeline can be dense. A simulation activity can make it tangible and memorable for your learners.

Quest: "The AI Apprentice"
Objective: Design a quest that simulates the LLM training pipeline, allowing students to experience the role of the AI and the human trainer at each stage.

Your Task:
Blueprint an activity with three phases, mapping to the training pipeline:

1. Phase 1: The Raw Mind (Pre-training Simulation):

   • Concept: How would you simulate the "internet dreamer" state of a base model?
   • Idea: Create a simple game where students are given the start of a sentence from a random, chaotic text source (e.g., mixed snippets of Wikipedia, Reddit, and old books). Their task is simply to complete the sentence in a way that seems stylistically consistent, without any goal of being helpful or correct. The scoring could reward fluency but not factual accuracy.

2. Phase 2: The First Lesson (SFT Simulation):

   • Concept: How would you simulate teaching the model to follow instructions?
   • Idea: Present students with "Instruction Cards," each containing a specific prompt (e.g., "Explain gravity to a five-year-old") and a target "ideal answer." Their task is to take a garbled, unhelpful output from the "Raw Mind" and edit it to match the ideal answer on the card. This simulates the supervised process of learning from examples.

3. Phase 3: The Trial of Preferences (RLHF Simulation):

   • Concept: How would you simulate the process of refining through human judgment?
   • Idea: Students now play the role of the human labeler. For a given prompt, they are shown two or three different responses generated by the "Phase 2" model. They must choose the best one according to a set of criteria (e.g., helpfulness, safety, clarity). Their choices contribute points to a hidden "Reward Model." In the final step of the quest, they can issue a new prompt and see the AI generate a response that reflects the collective preferences of the class.

Your Deliverable:
Write a brief outline (a few paragraphs) for this "AI Apprentice" quest. Describe the narrative wrapper, the core mechanic for each of the three phases, and how you would explain to your students how each phase connects to the real-world process of training an LLM.

Conclusion

In this lesson, we demystified the process of creating a modern LLM, breaking it down into a logical, three-stage pipeline.

Key Takeaways:

• LLM training is a sequential process: Pre-training builds general knowledge, Supervised Fine-Tuning teaches instruction-following, and RLHF refines the model based on human preferences.
• Pre-training creates a powerful but unaligned "base model" by training on massive amounts of internet text for next-word prediction.
• SFT adapts this base model into a helpful "assistant model" by training on a smaller, high-quality dataset of human-written prompt-response examples.
• RLHF addresses the limitations of SFT by using human preferences (ranking responses) to train a Reward Model, which then guides the final fine-tuning of the LLM using reinforcement learning.

We have now covered the model's architecture (Transformers) and its training pipeline. In our next lesson, we will zoom in on the most fundamental unit of data these models process: the token. We will explore how tokenization works and discuss its profound effects on model behavior, performance, and even cost.