Pre-training: Objectives, Data, and Compute

Hello! Welcome to the second module of our course, where we'll shift our focus from the architecture of LLMs to the process of creating them.

In the last lesson, we concluded our tour of the LLM landscape by comparing the major model families like GPT, Llama, and Claude. We established a key distinction between proprietary models accessed via APIs and open models you can run yourself.

Now, we'll begin to answer the question: how are these incredibly powerful base models built in the first place? This lesson tackles the first and most resource-intensive stage of that process, addressing the learning outcome: Describe the pre-training process: objectives, data, and compute requirements.

Think of pre-training as the equivalent of compiling a massive operating system from the entire public internet. It's a foundational, computationally immense process that happens infrequently but creates the powerful, general-purpose tool that all subsequent steps, like fine-tuning, will build upon.

1. What is Pre-training? The Core Objective

At its heart, pre-training is a self-supervised learning phase. The model isn't being taught to perform a specific task like translation or summarization. Instead, it's given one simple, overarching goal: to understand the statistical patterns of human language.

For the decoder-only models that dominate the modern landscape (like the GPT and Llama families), this goal is realized through an objective called autoregressive language modeling, which is a formal name for a simple idea: predicting the next word (or token). The model is shown a piece of text and its only job is to predict what token comes next. By doing this billions and billions of times over a vast dataset, it implicitly learns grammar, facts, reasoning abilities, and different styles of writing.

Andrej Karpathy offers a fantastic high-level explanation of this process. He frames pre-training as a form of "lossy compression" of the internet into a much smaller file of model parameters.

While next-word prediction is the dominant objective for generative models, it's worth knowing that other objectives exist, especially for different architectures:

• Masked Language Modeling (MLM): Used for encoder-only models like BERT. Instead of predicting the next token, the model predicts randomly masked (hidden) tokens within a sentence, allowing it to learn context from both left and right directions (i.e., bidirectionally).
• Span Corruption: Used by encoder-decoder models like T5. The model learns to reconstruct longer, corrupted "spans" of text rather than single tokens.

To see how these objectives map to their respective architectures, the following resource provides a concise overview.

2. The Three Pillars of Pre-training

Successfully pre-training a large language model depends on balancing three critical components: Data, Compute, and Scaling Laws.

Pillar 1: Data (The Fuel)

You can't create a knowledgeable model without an immense and high-quality source of knowledge.

• Scale & Source: Modern models are trained on trillions of tokens. This data is scraped from the public internet using services like Common Crawl, and is supplemented with curated sources like books, scientific papers, and code from GitHub. The wandb.ai article below describes a famous open dataset called "The Pile," which gives a great sense of the diversity required.

• Quality & Curation: Raw internet data is messy. A massive engineering effort goes into cleaning, filtering, and de-duplicating this data. High-quality data leads to better models, while garbage data can destabilize the training process. Increasingly, teams are also generating synthetic data using other powerful LLMs to create high-quality examples for specific abilities like reasoning or coding.

The diagram below shows the data processing pipeline for Meta's Llama 3.1, highlighting the importance of filtering, deduplication, and mixing different data sources.

For a practitioner's perspective on the crucial role of data, the following video is invaluable.

Pillar 2: Compute (The Engine)

Training on trillions of tokens requires a staggering amount of computational power.

• Hardware: Pre-training runs don't happen on a single machine. They require clusters of thousands of specialized processors like NVIDIA GPUs (e.g., A100s, H100s) or Google TPUs, connected by high-speed interconnects.
• Cost & Time: A single pre-training run can cost tens to hundreds of millions of dollars and take months to complete. As Karpathy noted, the $2 million cost for Llama 2 is now considered small scale.
• Parallelization: From a software engineering standpoint, this is a massive distributed systems challenge. To make this feasible, the model and data are split across the many GPUs in the cluster using various parallelization strategies (e.g., data parallelism, tensor parallelism, pipeline parallelism). This ensures all the hardware is used efficiently to shorten the training time from centuries to months.

The table below from the WandB article gives you a sense of the hardware scale involved for some well-known models.

Pillar 3: Scaling Laws (The Blueprint)

Given a fixed compute budget (e.g., $10 million), how should you allocate it? Should you build a larger model and train it on less data, or a smaller model trained on more data? This is where scaling laws come in.

Scaling laws are empirical rules that describe the relationship between model performance, model size, and the amount of training data. Research from labs like DeepMind (with their Chinchilla model) has shown that for optimal performance, the model size and the number of training tokens should be scaled up at roughly the same rate. Training a huge model on too little data is computationally inefficient, as is training a small model on too much data.

These laws allow researchers to run smaller, cheaper experiments to predict the performance of a much larger model, helping them make informed decisions before committing to a multi-million dollar training run.

The graphs in the following resource beautifully illustrate this power-law relationship.

3. A Quick Look at Continued Pre-training

Before we conclude, it's useful to know about a related technique: continued pre-training. Instead of starting from scratch with random weights, you can take an existing pre-trained model and continue training it on a new, specialized dataset (e.g., a medical or legal corpus). This is a cost-effective way to adapt a general model to a specific domain.

Conclusion

In this lesson, we've unpacked the monumental effort behind the pre-training of a large language model. It's a process defined by immense scale across data, compute, and cost.

Key Takeaways:

• Objective: The primary goal of pre-training is self-supervised learning, most commonly by predicting the next token in a vast corpus of text. This is how the model acquires its foundational knowledge and abilities.
• Data: Pre-training requires trillions of tokens of diverse, high-quality data, sourced from the internet, books, code, and more. Data curation and cleaning are critical, non-trivial engineering tasks.
• Compute: The process demands thousands of GPUs running for months, costing tens to hundreds of millions of dollars. It's a massive distributed computing challenge.
• Scaling Laws: These are crucial principles that guide researchers in efficiently allocating their finite compute budget between model size and data size to achieve optimal performance.

With this powerful, general-purpose "base model" created, the next step is to make it more useful and aligned with human intent.

Preview of Next Lesson: In our next lesson, we will explore Supervised Fine-Tuning (SFT). We'll see how developers take a pre-trained model and teach it to follow instructions and act as a helpful assistant by training it on a much smaller, curated dataset of high-quality prompt-response pairs.