Pre-training Foundational LLMs: Objectives and Data Scale

Recommended resources

The freely available resources this lesson will direct you to.

Introduction

In our previous lesson, we dissected the self-attention mechanism, understanding how it allows a Transformer model to weigh the importance of different tokens and build context-aware representations. We concluded with a crucial question: how do the model's parameters—the vast matrices of weights like $W^{Q}, W^{K}, W^{V}$ —come to possess the right values to perform these feats of contextual reasoning?

This lesson answers that question by diving into the world of pre-training. This is the foundational, and by far the most computationally intensive, stage of an LLM's life. We will explore the objectives that guide this training process and grapple with the colossal scale of data and computation required to forge a model like GPT-4 or LLaMA 3 from scratch. By the end, you will understand how a model learns general world knowledge before it is ever taught to be a helpful assistant.

The Lifecycle of an LLM: Starting with Pre-training

Modern large language models are typically created in a multi-stage process. We begin with a "base model" that is then refined for specific behaviors.

This diagram shows the three main stages of LLM development: (a) Pre-training, where a model learns from a massive, unlabeled corpus; (b) Instruction fine-tuning, where it learns to follow instructions from curated examples; and (c) Reinforcement learning, where its behavior is further refined based on human preferences.

Our focus today is on the first and most resource-intensive stage: Pre-training. The goal here is not to create a chatbot or an assistant, but to build a model with a comprehensive, general-purpose understanding of language and the world. The subsequent stages, which we will cover in the next lessons, adapt this general knowledge for specific tasks, like holding a conversation or following instructions.

To build an intuition for this distinction, Andrej Karpathy offers a clear explanation.

[1hr Talk] Intro to Large Language Models

In this segment of his "Intro to Large Language Models" talk, Andrej Karpathy distinguishes between the pre-training stage, which is about acquiring knowledge from vast quantities of text, and the fine-tuning stage, which is about aligning the model to a specific format or behavior, like being a helpful assistant.

Please watch the clip from fine-tuning vs. pre-training. Pay close attention to the concepts of "knowledge" versus "alignment" and the difference in the scale and quality of data used for each stage.

The Pre-training Objective: A Simple Task with Profound Consequences

So, what exactly is the model trying to do during pre-training? The dominant approach, particularly for models in the GPT family, is deceptively simple: predict the next token in a sequence. This is also known as Causal Language Modeling (CLM) or autoregressive language modeling.

The model is fed a massive amount of text from the internet, one sequence at a time. For each token in a sequence, its task is to predict the very next token. The model makes a prediction, compares it to the actual next token in the text, calculates the error (the loss), and adjusts its millions of parameters slightly to make a better prediction next time. This process is repeated for trillions of tokens.

Why is this simple objective so powerful? Karpathy provides an excellent explanation.

[1hr Talk] Intro to Large Language Models

This part of the talk explains how the simple objective of next-token prediction forces the model to learn complex patterns, grammar, facts, and reasoning, effectively compressing world knowledge into its parameters.

Watch the segment from how parameters are obtained. The key idea is that to be consistently good at predicting the next word in any context, the model has no choice but to internalize a vast model of the world.

As Karpathy explains, this process is a form of lossy compression. To accurately predict that the sequence "The first person on the moon was Neil..." is followed by "Armstrong", the model must have stored a representation of that fact within its parameters. Scaled across the entire internet, this simple objective compels the model to build an internal representation of grammar, syntax, facts, and even rudimentary reasoning abilities.

A Taxonomy of Objectives

While next-token prediction is dominant, it's not the only pre-training objective. The choice of objective is closely tied to the model's architecture (encoder-only, decoder-only, or encoder-decoder).

This diagram illustrates three common pre-training objectives and their corresponding model architectures. From top to bottom: Autoencoding (MLM) for encoder-only models like BERT, Autoregressive (CLM) for decoder-only models like GPT, and Sequence-to-Sequence (Span Corruption) for encoder-decoder models like T5.

Let's briefly survey these three main types:

Autoregressive (CLM): As we've discussed, this is used for decoder-only models (e.g., GPT series, LLaMA). The model can only see past tokens to predict the next one. This makes it inherently generative.
Autoencoding (MLM - Masked Language Modeling): This is used for encoder-only models (e.g., BERT). Instead of predicting the next token, the model predicts randomly masked tokens within the input. Since the encoder can see the full sequence (both left and right context), this creates powerful bidirectional representations, which are excellent for analytical tasks like classification or text understanding.
Sequence-to-Sequence (Span Corruption): This is used for encoder-decoder models (e.g., T5, BART). It's a hybrid approach where the model is given a corrupted input with several spans of text missing and is trained to generate the missing spans.

For a more detailed look at these objectives, the "Dive into Deep Learning" textbook provides a concise summary.

11.9. Large-Scale Pretraining with Transformers

This chapter from the d2l.ai textbook provides a technical overview of the pre-training objectives for the canonical models of each architectural type.

Please read the following three short sections to see how each objective works: In section "11.9.1. BERT: Bidirectional Encoder Representations from Transformers", read the subsection on pretraining BERT to understand masked language modeling. In section "11.9.2. T5: The Text-to-Text Transfer Transformer", read the subsection on pretraining T5 to understand span corruption. In section "11.9.3. GPT: Generative Pre-Training for Language Models", read the subsection on GPT to review autoregressive language modeling.

The Scale of Data and Compute

The effectiveness of pre-training is a direct function of scale. Foundational models are trained on datasets whose size is difficult to comprehend.

Let's start by exploring the data itself.

A Guide to Developing LLMs Part 1: Pretraining - DZone

This article provides an excellent breakdown of where the training data comes from, how it's processed, and the sheer volume involved in training modern LLMs.

Please read the entire section titled "Data". Pay close attention to the steps involved in preparing the data and the token counts for well-known datasets and models like LLaMA 3 and GPT-4.

The reading highlights several critical points:

The Source: A significant portion of the data comes from web scrapes like Common Crawl, which contains petabytes of raw web content.
The Curation: This raw data is unusable as-is. It undergoes a rigorous pipeline of cleaning, filtering (for quality and safety), deduplication, and re-weighting to create a final training corpus. This curation process is a crucial and often secret component of a model's success.
The Scale: The final datasets are measured in trillions of tokens. For context, the complete works of Shakespeare are less than a million words (perhaps 1.3 million tokens). LLaMA 3 was trained on 15 trillion tokens, equivalent to over 10 million copies of Shakespeare's entire collection.

This massive data requirement naturally leads to an equally massive need for computational power. This relationship is formalized by Scaling Laws.

Scaling Laws: A Predictable Path to Better Models

One of the most significant discoveries in the field of LLMs is that their performance improves in a predictable way as you increase three factors: model size (number of parameters), dataset size (number of tokens), and the amount of compute used for training. This relationship follows a power law, meaning that on a log-log plot, the model's loss (a measure of its error) decreases linearly as these factors increase.

11.9. Large-Scale Pretraining with Transformers

This section from d2l.ai visually demonstrates the power-law scaling relationship that underpins the development of ever-larger LLMs.

Please read the section titled "Scalability". Focus on the graphs that show performance improving smoothly with increases in model size, dataset size, and compute.

This predictability is what fuels the "gold rush" for larger models and bigger GPU clusters. However, simply making the model bigger isn't the most efficient strategy. A pivotal discovery, detailed in the Chinchilla paper, refined our understanding of these laws.

The key finding was that for a fixed computational budget, the best performance is achieved not by training the largest possible model, but by training a smaller model on proportionally more data. Previous models were often "undertrained"—they were so large that they hadn't seen enough data to reach their full potential within the given compute budget. Chinchilla proposed an optimal token-to-parameter ratio of roughly 20:1.

This video provides an excellent summary of the Chinchilla paper and its implications.

Chinchilla Explained: Compute-Optimal Massive Language Models

This video, "Chinchilla Explained", breaks down the paper's core findings about compute-optimal training and why training data scale is just as important as parameter scale.

Please watch the following key segments: Introduction: The narrator sets up the problem, contrasting the old approach (scaling parameters faster than data) with Chinchilla's finding. Data Scale Comparison: A quick comparison showing Chinchilla using vastly more tokens than previous models. Oversized and Undertrained: This explains the core implication—that many large models were inefficiently trained. The Chinchilla Model: The video explains how the Chinchilla model itself was trained using these optimal ratios, leading to superior performance with fewer parameters.

The Unthinkable Cost of a Foundational Model

Combining the immense data requirements with the compute-optimal scaling laws results in a training process of astronomical cost. The DZone article you read earlier provides a sobering, concrete estimate for training a state-of-the-art model like LLaMA 3.

A Guide to Developing LLMs Part 1: Pretraining - DZone

We will revisit this article to see the final tally of what it takes to train a frontier model.

Read the section titled "Training a SOTA Model". This puts concrete numbers to the entire pre-training enterprise.

The numbers are staggering:

Data: 15.6 trillion tokens
Hardware: 16,000 NVIDIA H100 GPUs
Time: ~70 days of continuous training
Cost: ~$75 million
Environmental Impact: ~4,400 tons of CO₂, equivalent to 2,000 round-trip flights between New York and London.

This immense cost is why foundational models are created by only a handful of large, well-funded organizations.

Conclusion

In this lesson, we've journeyed through the demanding process of pre-training a large language model. You now have the vocabulary and conceptual framework to discuss this foundational stage with technical depth.

Here are the key takeaways:

Pre-training's Goal: To imbue a model with general world knowledge and linguistic ability, not to make it a helpful assistant. This is achieved through self-supervised objectives on unlabeled data.
Primary Objective: The most common objective is autoregressive next-token prediction, which forces the model to "compress" knowledge about the world into its parameters. Other objectives like MLM and span corruption exist for different architectures.
Immense Scale: Foundational models are pre-trained on trillions of tokens of carefully curated text, sourced primarily from web scrapes like Common Crawl.
Scaling Laws: Performance is a predictable function of model size, data size, and compute. The Chinchilla paper demonstrated that for a fixed compute budget, scaling data is just as critical as scaling model parameters.
Prohibitive Cost: The combination of data and compute scale makes pre-training a frontier model an endeavor costing tens of millions of dollars and significant energy.

We have built our "base model," a vast repository of knowledge. But in its current state, it's more like an internet document completion engine than a useful tool. In our next lesson, we will explore supervised fine-tuning, the first step in "aligning" this raw intelligence to become a helpful, instruction-following assistant.