Introduction
In our previous lesson, we examined the predictable, quantitative improvements in large language models as described by scaling laws. We saw how increasing compute, parameters, and data leads to a smooth, measurable decrease in the model's loss. This gives us a powerful framework for resource allocation and predicting performance on the training objective.
Today, we turn to a far more surprising and less predictable consequence of scaling. As models grow, they don't just get incrementally better at the same things; they suddenly acquire entirely new abilities that smaller models completely lack. This lesson delves into the fascinating phenomenon of emergent capabilities, where quantitative increases in scale give rise to sharp, qualitative leaps in model behavior. We will define what constitutes an emergent ability, look at concrete examples, and explore the leading scientific explanations for why they occur. This moves us from the "how much better" of scaling laws to the "what else can it do" that defines the cutting edge of AI research.
What Are Emergent Abilities?
At its core, the idea of emergence is not new and comes from fields like physics and biology. It describes situations where a system's properties cannot be understood merely by studying its individual components; instead, these properties arise from the complex interactions of those components at a certain scale. The Nobel prize-winning physicist Philip W. Anderson famously summarized this as "More is Different."
To build your intuition, let's start with a video that uses several clear analogies to introduce this concept in the context of LLMs.
The Emergent Abilities of LLMs - why LLMs are so useful
This video from AssemblyAI provides an excellent high-level introduction to emergent abilities, drawing parallels to phenomena like a child learning to draw and phase transitions in physics.
Please watch the introduction from the beginning to get a feel for the concept. Then, watch the segment on emergence in physics, which introduces the powerful analogy of phase transitions, like water turning to ice. This will be a useful mental model throughout the lesson.
As the video suggests, a small amount of uranium is harmless, but a critical mass triggers a nuclear reaction. A few water molecules behave one way, but a large collection undergoes a phase transition into ice at 0°C. LLMs exhibit a similar behavior.
The seminal paper "Emergent Abilities of Large Language Models" by Wei et al. (2022) provided a formal definition for this phenomenon in the context of language models. Let's examine their work to get a precise understanding.
[PDF] Emergent Abilities of Large Language Models - OpenReview
This is the foundational research paper that defined and popularized the study of emergent abilities in LLMs. Reading the introduction and definition will give you the formal framework used by researchers.
Read the Abstract to get the high-level summary. Then, focus on the Introduction, which connects LLM emergence to the broader scientific concept. Finally, carefully read Section 2, Emergent Abilities Definition, for the specific definition we will use.
To summarize the key points from the paper:
- An ability is emergent if it is not present in smaller models but is present in larger models.
- Crucially, these abilities cannot be predicted by simply extrapolating the performance of smaller models.
- When plotted on a graph of performance vs. model scale, emergent abilities show a characteristic "hockey stick" curve: performance is near-random for a range of smaller models, and then, after crossing a critical scale, it increases sharply to well above random. This sharp change is often called a phase transition.
Examples of Emergence
Now that we have a definition, let's look at what this phenomenon looks like in practice. The most compelling evidence comes from plots showing the performance of different model families across a wide range of scales on various tasks.
The video below, presented by Jason Wei (the lead author of the paper we just looked at), walks through the canonical way these abilities are visualized and provides several concrete examples.
Stanford CS25: V2 I Emergent Abilities and Scaling in LLMs
In this lecture from the Stanford CS25 course, Jason Wei explains how emergent abilities are measured and provides several clear examples.
Watch the segment from how we look at plots for emergence. Pay close attention to the x-axis (model scale, often in training FLOPs) and y-axis (task performance/accuracy). Notice the flat, near-random performance of smaller models and the sudden takeoff at a certain scale.
The paper "Emergent Abilities of Large Language Models" catalogs dozens of such tasks. These include:
- Multi-step Arithmetic: Performing 3-digit addition or 2-digit multiplication.
- Massive Multi-task Language Understanding (MMLU): Answering questions from 57 different subjects, from high school math to professional law.
- Word Unscrambling: Guessing a word from its scrambled letters.
In all these cases, models below a certain size (e.g., ~10 billion parameters) perform no better than random guessing. Then, almost as if a switch is flipped, performance rapidly climbs in larger models.
Emergence of Prompting Strategies: Chain-of-Thought
Perhaps one of the most significant emergent capabilities is not a task, but a technique. Chain-of-thought (CoT) prompting is a method where the model is prompted to "show its work" by generating intermediate reasoning steps before giving a final answer.
This technique dramatically improves performance on complex reasoning tasks. However, the ability to successfully follow a chain of thought is itself emergent. Applying CoT prompting to smaller models often degrades performance, as they fail to produce coherent reasoning. It's only with sufficiently large models that the technique "unlocks" and provides substantial benefits.
Let's return to Jason Wei's lecture for a clear explanation and demonstration.
Stanford CS25: V2 I Emergent Abilities and Scaling in LLMs
This part of the lecture provides an excellent overview of Chain-of-Thought prompting and demonstrates that the effectiveness of this technique is an emergent property of scale.
Watch from the section on Chain of Thought prompting, where the concept is introduced and demonstrated. Then, watch the crucial follow-up from here's another example of emergence, which shows the performance plot confirming that CoT only helps for large enough models.
Explaining Emergence: Illusion or Phase Transition?
The existence of these sharp performance jumps is undisputed, but the underlying cause is a topic of intense scientific debate. Why do these qualitative shifts happen? There are two main hypotheses.
Hypothesis 1: An Artifact of the Metric
One compelling argument is that emergence is an "illusion" created by our choice of evaluation metrics. Many tasks, particularly complex ones, require multiple sub-steps to be completed correctly. If we only give credit for a perfect final answer (a non-linear or discontinuous metric), we might miss smooth, continuous improvements happening at a lower level.
Imagine a task that requires 5 sequential reasoning steps. Let be the probability that the model correctly performs a single step. To solve the entire task, the model must get all 5 steps right. Assuming independence, the total probability of success is .
Now, consider what happens as the model scales:
- A small model might have . Its success rate is (near random).
- A medium model might improve to . Its success rate is .
- A large model might achieve . Its success rate is .
The underlying capability () improves smoothly, but the measured success rate () shows a sharp, non-linear jump that looks like emergence.
The following diagram from a recent research paper beautifully illustrates this idea.
This hypothesis suggests that what we call "emergence" might simply be the macroscopic result of continuous, microscopic improvements crossing a threshold defined by our measurement system.
Hypothesis 2: A Genuine Phase Transition
The alternative view, which you encountered in the AssemblyAI video's analogy of water freezing, is that emergence reflects a genuine phase transition within the model's internal dynamics. This perspective draws from the study of complex adaptive systems.
The core idea is that an LLM is a complex system of billions of interacting components (the parameters).
- At smaller scales, these interactions are not rich enough to form stable, large-scale structures. The model can only capture simple, local statistical patterns.
- At a critical scale, the density and strength of these interactions cross a threshold. This allows for self-organization, where higher-order structures—representing abstract concepts, reasoning chains, and grammatical rules—can form and stabilize.
From this viewpoint, the model's behavior undergoes a true qualitative change. The system is no longer just memorizing sequences; it's developing an internal, compositional model of language and the world. This allows it to generalize in ways that lead to new capabilities. Recent theoretical work, such as the paper "A non-ergodic framework for understanding emergent capabilities in Large Language Models" by Javier Marín, formalizes this by modeling LLMs as non-ergodic systems where path-dependent, historical context allows for the formation of novel states (capabilities) that were not accessible before.
This debate is far from settled. It is likely that both explanations contribute. Some emergent phenomena may be artifacts of metrics, while others may reflect more fundamental reorganizations of the model's internal representations.
Conclusion
In this lesson, we have journeyed beyond the smooth, predictable world of scaling laws into the surprising realm of emergent capabilities. We have seen that simply making models bigger can cause them to spontaneously develop new skills, a phenomenon that has profound implications for the future of AI.
Here are the key takeaways:
- Emergence Defined: An emergent ability is a capability that appears in large models but not in smaller ones, in a way that is not predictable by simple extrapolation. This often manifests as a "hockey stick" performance curve.
- Examples of Emergence: Many capabilities, from arithmetic to complex reasoning, have been shown to be emergent. Furthermore, techniques like Chain-of-Thought prompting are themselves emergent, only becoming effective at a sufficient scale.
- Two Competing Explanations: The cause of emergence is an active area of research. One hypothesis posits it's an illusion created by non-linear metrics that mask smooth underlying improvements. The other, more profound hypothesis suggests it's a genuine phase transition in the model's internal dynamics, akin to self-organizing complex systems.
So far, we have discussed emergence in general terms. In our next lesson, we will perform a deep dive into what is arguably the most important and powerful emergent capability of all: in-context learning. This is the remarkable ability of a model to learn a new task from just a few examples provided in its prompt, without any changes to its weights.