Scaling Laws for Neural Network Performance

Introduction

In our last module, we explored the array of parallelism techniques—data, pipeline, and tensor—that provide the mechanisms for training enormous models across distributed hardware. We now have the tools to answer "How can we train a model with a trillion parameters?" But this ability immediately raises a more strategic set of questions. From your DevOps perspective, you know that having resources is one thing; allocating them optimally is another entirely.

Imagine you are given a fixed budget—say, access to a large GPU cluster for one month. How do you decide what to build? Should you train a medium-sized model for many epochs on a vast dataset, or a colossal model for just a few epochs on a smaller one? Is there a predictable relationship between the resources you invest and the performance you can expect?

This lesson delves into the foundational principles that guide these critical decisions: scaling laws. We will focus on the seminal 2020 paper from OpenAI, "Scaling Laws for Neural Language Models" by Jared Kaplan et al., which first demonstrated that the chaotic process of deep learning can be surprisingly predictable. These empirical laws revealed that a model's performance, as measured by its test loss, improves smoothly as a power-law function of model size, dataset size, and the total compute used for training.

Our goal is to explain these relationships and understand their profound implication for training large language models: the existence of a compute-optimal training strategy.

The Engineering of AI Progress: Scaling as a Predictable Science

Before the work on scaling laws, improving language models often felt like a series of disjointed research efforts—a new architecture here, a better optimizer there. Progress was hard to predict. The Kaplan paper helped shift this paradigm, suggesting that progress in AI capabilities could be more like an engineering discipline than a quest for singular breakthroughs, akin to building a better steam engine rather than proving the Riemann Hypothesis. The idea is that if performance scales predictably, we can chart a course for progress simply by increasing our investment in the right resources.

The core discovery is that key performance metrics follow power laws. A power-law relationship between two variables, say loss $L$ and model size $N$ , means that $L$ is proportional to $N$ raised to some power. On a graph with logarithmic scales on both axes, this relationship appears as a straight line, making it possible to extrapolate performance far beyond what has been empirically measured.

Let's begin by watching Jared Kaplan, the lead author of the paper, introduce this perspective and the main empirical findings.

Neural Scaling Laws and GPT-3

In this talk, "Neural Scaling Laws and GPT-3", Jared Kaplan frames the problem and presents the core results of his team's work.

First, watch the introductory segment from the big picture, where he contrasts the "engineering" versus "genius insight" view of AI progress. Then, watch from the main findings, where he explains the three key log-log plots that form the basis of the scaling laws. Pay close attention to the x-axes: model parameters, dataset size, and compute.

The plots you just saw are the heart of this lesson. They show that test loss decreases predictably as a power law when we scale up three key factors, as long as the other two are not a bottleneck.

These three plots illustrate the power-law relationship between test loss and (from left to right) compute budget, dataset size, and the number of non-embedding model parameters. The straight lines on these log-log axes demonstrate the predictability of scaling.

The Three Pillars of Scaling: N, D, and C

The Kaplan paper isolates three variables and provides a scaling law for each one. Let's examine them individually. The foundational paper itself provides the most precise definitions.

Scaling Laws for Neural Language Models

This is the original paper by Kaplan et al. (2020). We will use it as our primary reference for the specific formulas and exponents.

Please read the following sections: The Abstract for a high-level overview. The "Summary" in Section 1.1, starting here, for a concise list of the main discoveries. The "Summary of Scaling Laws" in Section 1.2, starting here, to see the explicit equations. Finally, glance at the "Summary of Power Laws" tables in Appendix A, located here, which provide a handy reference for all the parameters.

As you've seen in the paper, the test loss $L$ can be modeled as a function of model parameters ( $N$ ), dataset size ( $D$ ), or compute ( $C$ ).

1. Scaling with Model Size (N)

When trained on a very large dataset (so data is not a bottleneck), the model's performance is limited by its own capacity. The relationship is given by:

L (N) = (\frac{N _{c}}{N})^{α_{N}}

$N$ : The number of non-embedding parameters in the model. This is a crucial detail; the authors found that including embedding parameters, which scale with vocabulary size, obscured the clean power-law trend.
$α_{N} \approx 0.076$ : The scaling exponent for model size. The small value indicates that while loss does decrease, it does so slowly. Doubling model size reduces loss by a factor of $2^{- 0.076}$ , which is only about a 5% reduction.
$N_{c} \approx 8.8 \times 1 0^{13}$ : A constant representing a characteristic model size.

This law tells us that bigger models perform better, and quantifies by exactly how much.

2. Scaling with Dataset Size (D)

When using a very large model (so capacity is not a bottleneck), performance is limited by the amount of data it's trained on. The relationship is:

L (D) = (\frac{D _{c}}{D})^{α_{D}}

$D$ : The number of tokens in the training dataset.
$α_{D} \approx 0.095$ : The scaling exponent for dataset size.
$D_{c} \approx 5.4 \times 1 0^{13}$ : A constant representing a characteristic dataset size.

Notice that $α_{D} > α_{N}$ . This means that, when considered in isolation, a percentage increase in data yields a larger percentage decrease in loss than the same percentage increase in model size. However, this does not account for the cost of processing that data.

3. Scaling with Compute (C)

Ultimately, both model size and dataset size are constrained by the total compute budget. Compute, measured in FLOPs (Floating Point Operations), is the most fundamental resource. For a Transformer, the total training compute can be approximated as $C \approx 6 \times N \times D$ . The factor of 6 comes from the fact that the backward pass is roughly twice as computationally expensive as the forward pass (1 FLOP per parameter for forward, 2 for backward, totaling 3, and this is doubled to account for both weights and activations).

When the compute budget is allocated optimally between $N$ and $D$ , the relationship is:

L(C_{\text{min}}) = \left(\frac{C_{\text{min_c}}}{C_{\text{min}}}\right)^{\alpha_C}

$C_{min}$ : The optimally allocated compute budget.
$α_{C} \approx 0.050$ : The scaling exponent for compute.
$C_{\text{min_c}} \approx 3.1 \times 10^8$ : A characteristic compute budget in PetaFLOP-days.

The exponent $α_{C}$ is smaller than both $α_{N}$ and $α_{D}$ . This makes intuitive sense: since compute is spent on both increasing $N$ and processing $D$ , its effect on loss is "diluted" across both factors.

For a clearer understanding of how these individual laws were derived and what they imply, this blog post provides an excellent summary.

Kaplan Scaling Laws: Predicting Language Model Performance

This article breaks down each of the three scaling laws with clear explanations and code snippets to illustrate the relationships.

Please read from the start of the article, covering the sections on the experimental foundation and the individual scaling laws for parameters, data, and compute.

The Main Event: Optimal Compute Allocation

The individual scaling laws are interesting, but their combination leads to the paper's most influential and controversial conclusion. Given that $C \approx 6 N D$ , a fixed compute budget $C$ forces a tradeoff between $N$ and $D$ . What is the optimal way to spend that budget to achieve the lowest possible loss?

The Kaplan paper answers this by analyzing how to allocate an increasing compute budget. Their finding was stark: you should prioritize making the model larger much more aggressively than training it on more data.

The optimal allocation rules they derived are:

Optimal Model Size: $N_{opt} \propto C^{0.73}$
Optimal Dataset Size: $D_{opt} \propto C^{0.27}$

This means that if you increase your compute budget by a factor of 10, you should increase your model size by a factor of $1 0^{0.73} \approx 5.4$ , but your dataset size by only $1 0^{0.27} \approx 1.9$ .

The underlying reason is that larger models are more sample-efficient—they learn more from each token they see. Therefore, it's better to use your compute to build a larger model and train it for fewer steps than to train a smaller model to convergence on a massive dataset. This principle directly guided the development of models like GPT-3, which used 175 billion parameters but was trained on "only" 300 billion tokens (a ratio of fewer than 2 tokens per parameter).

The following chart from the paper visualizes this allocation.

This chart illustrates how an increasing compute budget (x-axis) should be allocated. The vast blue area shows that most of the additional compute should be spent on increasing Model Size, while the orange and green areas (related to data via Batch Size and Serial Steps) grow much more slowly.

Let's watch a final clip from the Stanford lecture that drives this point home.

Stanford CS336 Language Modeling from Scratch | Spring 2025 | Lecture 9: Scaling laws 1

This segment of the Stanford lecture on scaling laws introduces the joint data-model scaling law and explains how it is used to determine the optimal allocation of resources.

Please watch from this section. The speaker sets up the core question—more data or bigger models?—and then introduces the joint scaling law that provides the answer.

Conclusion

In this lesson, we've moved from the mechanisms of scaling to the principles that govern it. The Kaplan et al. scaling laws established a quantitative and predictive framework for language model development, demonstrating that performance improves smoothly with scale.

Here are the key takeaways:

Predictable Performance: Language model test loss scales as a power law with model size ( $N$ ), dataset size ( $D$ ), and compute ( $C$ ). This means performance on a log-log plot is a straight line, enabling extrapolation.
The Three Laws: The paper provides specific exponents for each factor: $L \propto N^{- 0.076}$ , $L \propto D^{- 0.095}$ , and $L \propto C^{- 0.050}$ , when other factors are not bottlenecks.
Optimal Allocation: For a fixed compute budget, Kaplan's analysis concluded that it is optimal to allocate most of the resources to increasing model size ( $N \propto C^{0.73}$ ) rather than dataset size ( $D \propto C^{0.27}$ ).
Sample Efficiency: This allocation strategy stems from the finding that larger models are more sample-efficient, extracting more information per token than smaller models.

These findings were transformative, providing a recipe for how to spend billions of dollars on training runs. However, science is an iterative process. While the Kaplan laws provided the first rigorous answer, they were not the final word.

In our next lesson, we will explore the Chinchilla scaling laws from DeepMind. This subsequent work re-evaluated the optimal allocation strategy and came to a different conclusion, suggesting that models like GPT-3 were, in fact, significantly undertrained for their size. This will paint a richer picture of the complex interplay between model, data, and compute.

Recommended resources