Pre-training and Fine-tuning LLMs

Introduction

In our last session, we explored the KV cache, a crucial optimization that makes the inference process of large language models efficient. We saw how models like GPT generate text one token at a time by caching and reusing key-value pairs, transforming a quadratically complex task into a linear one.

Today, we shift our focus from how LLMs generate text to how they are created. Our learning outcome is to describe the pre-training and fine-tuning paradigm for large language models. This two-stage process is the fundamental recipe behind virtually all modern LLMs, from open-weights models like Llama 3.1 to proprietary systems like GPT-4o. You'll learn how a model first acquires vast world knowledge and then is taught to be a helpful, instruction-following assistant.

1. The Two-Act Play: Pre-training and Fine-tuning

At its core, a large language model is surprisingly simple. As influential AI researcher Andrej Karpathy puts it, you can think of an LLM as just two files: one containing the model's parameters (the "weights") and another containing the code to run them. The billion-dollar question is: where do these parameters, which weigh hundreds of gigabytes, come from?

The process of creating these parameters is best understood as a two-act play.

1. Act I: Pre-training. This is a computationally massive, expensive process where the model learns the fundamentals of language, reasoning, and world knowledge by processing a huge portion of the internet. The result is a base model.
2. Act II: Fine-tuning. This is a much cheaper, faster process where the base model is "aligned" to be a helpful assistant. This is where it learns to follow instructions and converse in a question-and-answer format. The result is an assistant model.

Let's begin with a high-level overview of this entire process from Andrej Karpathy.

The following diagram visually summarizes the pipeline Karpathy described. It shows a randomly initialized model being pre-trained on vast internet data to produce a "Pre-Trained Model," which is then fine-tuned on specialized data to become a "Fine-Tuned Model."

Now, let's deconstruct each of these two acts.

2. Act I: Pre-training - Building the Knowledge Base

The goal of pre-training is to imbue the model with a comprehensive understanding of language, grammar, facts, and reasoning patterns. It's where the model learns that "Paris" is the capital of "France" and how to structure a Python function.

This is achieved through a simple-sounding but powerful objective: next-token prediction.

As the article explains, pre-training is self-supervised. The model learns from raw, unlabeled text because the label is implicitly the next word in the sequence. This is what allows it to be trained on colossal datasets like the Common Crawl, which is a snapshot of a significant portion of the public web.

To truly master next-token prediction on such diverse text, the model is forced to internalize a vast amount of world knowledge within its parameters. Predicting the word that follows "The first person on the moon was Neil..." requires knowing who Neil Armstrong is. This is why the seemingly simple objective of next-token prediction results in such knowledgeable models.

However, this process isn't perfect.

A base model is like a brilliant but unhelpful savant. It has read the library but will only respond by continuing the text you've started, often mimicking the style of an internet document rather than answering a direct question. To make it useful, we need to fine-tune it.

3. Act II: Fine-tuning - Aligning with Human Intent

Fine-tuning is the process of adapting the pre-trained base model to perform specific tasks or adhere to certain behaviors, like being a helpful and harmless assistant. This stage is less about learning new knowledge and more about learning a new style of interaction. The primary technique used here is Supervised Fine-Tuning (SFT).

By training on thousands of these curated examples, the model learns to adopt the persona of an assistant. It understands that input text is an "instruction" to be followed, and its output should be a helpful "response."

However, SFT has its limits. It can be difficult and expensive to write high-quality, detailed responses for every possible query. Sometimes, it's much easier for a human to compare two different answers and say which one is better. This insight leads to a further refinement step, often called Reinforcement Learning from Human Feedback (RLHF).

The core idea of RLHF is to:

1. Train a separate "reward model" that learns to predict which of two responses a human would prefer.
2. Use that reward model to provide a score (a "reward") to the LLM's outputs during further training.
3. Use Reinforcement Learning algorithms to update the LLM's policy to maximize this reward score.

The key takeaway is that RLHF allows us to steer the model towards human preferences in a more scalable way than writing perfect examples for every situation. Andrej Karpathy offers a great intuition for why this works.

The combination of SFT and preference-based methods like RLHF is what transforms a raw base model into the polished, helpful AI assistants we interact with daily. The entire process is iterative; developers collect data on where the assistant fails, integrate that data back into the fine-tuning set, and repeat the process to continuously improve the model.

Conclusion

In this lesson, we have demystified the high-level process of creating a large language model. You now understand that it's not a single monolithic training run but a carefully staged paradigm.

Key Takeaways:

• Two-Phase Paradigm: LLM development consists of pre-training (knowledge acquisition) and fine-tuning (behavioral alignment).
• Pre-training: Uses a self-supervised, next-token prediction objective on massive, uncurated internet-scale text. This creates a knowledgeable but unaligned base model. It is extremely expensive and done infrequently.
• Fine-tuning (Post-training): Adapts the base model to be a helpful assistant.
  • Supervised Fine-Tuning (SFT) is the first step, where the model learns the instruction-following format from high-quality, human-curated prompt-response pairs.
  • Reinforcement Learning from Human Feedback (RLHF) can further refine the model by training it to optimize for human preferences, which are often easier to collect (via comparisons) than perfect examples.
  • This stage is computationally cheaper and is iterated on frequently to improve the model.

Preview of the Next Lesson:
This lesson concludes our module on the foundational concepts of Transformer architecture and LLM training. We are now ready to bridge the gap between these basics and the state-of-the-art models mentioned in our syllabus.

In the next lesson, we will begin the "Modern Transformer Components" module. We will start by examining RMSNorm and the SwiGLU activation function—two simple but powerful architectural modifications that are standard in models like Qwen 2.5 and Llama 3, improving their training stability and overall performance.