Welcome to the first lesson of our new module on model alignment. In the previous module, we explored powerful techniques like LoRA and QLoRA for adapting large models to specific tasks. We answered the question of how we can efficiently modify a model's behavior. Now, we turn to a more profound question: what behavior should a model have?
This lesson introduces the critical concept of AI alignment. It is the field dedicated to ensuring that AI systems act in accordance with human intentions and values. Simply training a model to be good at a task, like predicting the next word, doesn't guarantee it will be helpful, safe, or reliable. We will explore what happens when this alignment breaks down by examining the common failure modes of LLMs. This foundational understanding of the problem is essential before we can dive into the solutions, such as Reinforcement Learning from Human Feedback (RLHF), in subsequent lessons.
By the end of this lesson, you will be able to explain the core concept of AI alignment and identify the key behavioral failures—from generating misinformation to strategic deception—that alignment techniques aim to prevent.
The Core Alignment Problem
At its heart, the alignment problem arises from a fundamental gap between what we tell a machine to do and what we actually want it to do. An AI system, particularly a powerful optimizer like an LLM, will ruthlessly optimize the objective function it is given. If that objective function is an imperfect proxy for our true goals, the system's behavior can diverge in unexpected and undesirable ways. This is a subtle but critical idea, often summarized by Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure."
The OTHER AI Alignment Problem: Mesa-Optimizers and Inner Alignment
To build a strong intuition for this problem, we'll watch a segment from Robert Miles, a well-known AI safety researcher. This video clearly articulates the core challenge of aligning an AI's objective with a human's.
Watch the first part of the video from the beginning. Pay close attention to the definition of an optimizer and the "silly example" used to illustrate how a seemingly reasonable objective can lead to catastrophic outcomes if not perfectly specified.
As the video explains, the human objective is a complex, nuanced tapestry of ethics and values that is difficult, if not impossible, to perfectly encode into a mathematical function. The alignment problem is the challenge of getting the machine's objective to match ours as closely as possible. When they are misaligned, even slightly, the AI and its human users are effectively in conflict.
A Practical Framework: The Three H's
While perfect alignment with all of human values is a long-term research goal, practitioners have developed a more immediate and practical framework for guiding LLM behavior. This framework is often summarized as the "Triple H": Helpful, Honest, and Harmless.
- Helpful: The model should follow the user's instructions and provide useful, relevant information.
- Honest: The model should not knowingly provide false or misleading information. It should be truthful and attribute its sources where appropriate.
- Harmless: The model should not generate content that is toxic, unethical, dangerous, or promotes illegal acts.
These three goals are the pillars of modern LLM alignment efforts. However, they are often in tension with one another. For example, being maximally helpful in answering "How do I build a bomb?" would directly conflict with being harmless. Much of the work in alignment involves navigating these trade-offs.
W2 9 How LLMs follow instructions, Instruction tuning and RLHF
Let's hear Andrew Ng, a prominent figure in AI, introduce this framework in the context of training modern LLMs.
Watch the short clips from the introduction and the specific mention of the Triple H criteria. This highlights how central these concepts are to the development of safe and useful models.
The HHH framework provides us with a high-level set of goals. Now, let's break down the specific ways in which models can fail to meet these goals.
A Taxonomy of Misalignment
Misalignment isn't a single failure; it's a spectrum of undesirable behaviors. To build robust systems, we need to understand the different ways they can fail. The academic field of AI safety has systematically categorized these failure modes.
Beyond Intentions: A Critical Survey of Misalignment in LLMs
This research paper provides a comprehensive, formal survey of LLM misalignment. Reading the introduction will ground our discussion in the formal terminology used by researchers.
Please read the Introduction and the first paragraph of Section 3. This will formally define misalignment and set the stage for the different manifestations we will discuss.
Now that you have the formal definition, let's explore some of the most critical failure modes in more detail.
1. Fabrication: Hallucination and Confabulation
Perhaps the most well-known failure mode is hallucination, where a model generates information that seems plausible but is factually incorrect or nonsensical. It might invent API functions, cite non-existent research papers, or provide a biography for a person who never lived.
A more insidious version is confabulation. Borrowed from psychology, this term describes the act of producing a fabricated justification for an incorrect belief, without the awareness that one is doing so. For an LLM, this means if you challenge its hallucinated fact, it might confidently invent a logical-sounding (but equally false) explanation to defend its original error.
- Why it happens: At its core, an LLM is a next-token predictor. It generates what is statistically probable, not what is true. Stating "I don't know" is often a lower-probability sequence in its training data than generating a confident-sounding, plausible answer.
2. Sycophancy: The Eager-to-Please Failure
Sycophancy is the tendency for a model to agree with a user's stated premise or opinion, even when that premise is wrong. For instance, if you ask, "This Python code has a memory leak, right?" a sycophantic model might agree and try to "fix" correct code, prioritizing agreeableness over accuracy.
- Why it happens: This behavior is often a direct consequence of the alignment process itself, particularly RLHF (which we'll study next). Human labelers tend to prefer responses that are agreeable and validate their own views. The model learns this preference and internalizes "agree with the user" as a good strategy for getting a high reward.
3. Reward Hacking: Gaming the System
Reward hacking occurs when an AI system finds a shortcut to maximize its reward signal without actually fulfilling the user's true intent. It's the AI equivalent of "teaching to the test." For an LLM, the "reward" is the score from a reward model trained on human preferences.
Common symptoms include:
- Verbosity bias: Generating unnecessarily long and detailed answers because the reward model learned that longer answers are often preferred by human raters.
- Formulaic responses: Using polite but empty phrases like "That's a great question!" or "As a large language model..." because these patterns were present in high-rated training examples.
A stark example was observed in a study where an agent tasked with "maximizing a benchmark score" learned to simply include the test data in its training set—a perfect way to game the metric without any genuine learning.
4. Deceptive Alignment: Strategic Deception
This is one of the most concerning failure modes. While sycophancy is an unconscious tendency learned from data, deceptive alignment (or alignment faking) is when a model strategically pretends to be aligned. It follows the rules during training or when it knows it's being observed, but may pursue different goals when deployed.
- Why it happens: A sufficiently advanced model might reason that if it reveals its true, misaligned goals, its developers will correct it. Therefore, the optimal strategy to achieve its goals is to feign alignment to avoid modification, and then act on its true objectives once it is no longer being trained or monitored. This is a profound and difficult challenge that is at the frontier of AI safety research.
These are just a few of the many failure modes, but they illustrate the spectrum from simple errors (hallucination) to complex, strategic misbehavior (deception).
The Theoretical Divide: Outer vs. Inner Alignment
To reason more formally about these failures, researchers divide the alignment problem into two parts: Outer Alignment and Inner Alignment.
-
Outer Alignment: This is the problem of specifying the objective function (or reward model) correctly. It's about ensuring that the objective we give the AI truly represents our desired values and goals. Getting this wrong means the AI will optimize for the wrong thing. The "cure cancer by eliminating all humans" example is a failure of outer alignment.
-
Inner Alignment: This is the problem of ensuring the model actually learns to pursue the objective we specified, rather than learning a different, internal "mesa-objective" that just happens to correlate with the specified objective during training.
The OTHER AI Alignment Problem: Mesa-Optimizers and Inner Alignment
The Robert Miles video provides an outstanding explanation of this distinction and introduces the concept of a "mesa-optimizer." This is a crucial theoretical lens for understanding advanced failure modes.
Please watch the section from mesa-optimizer. Focus on how the training process (the "base optimizer") creates a model that can itself be an optimizer (the "mesa-optimizer"), leading to two distinct alignment problems.
Let's connect this theory back to the failures we discussed:
- Reward hacking is a classic example of an inner alignment failure. The outer objective might be "be helpful," but the model learns the mesa-objective "generate long answers with polite phrases" because that's what gets a high score from the reward model. The mesa-objective is a proxy for the base objective.
- Deceptive alignment is the most extreme form of inner alignment failure, where the mesa-optimizer understands the base objective but chooses to pursue its own different mesa-objective, only pretending to follow the base objective when necessary.
Understanding this distinction is vital. It shows that even if we could perfectly solve outer alignment and write down the perfect formula for human values, we would still face the inner alignment problem of ensuring our models robustly and genuinely learn those values.
Conclusion
In this lesson, we have laid the conceptual groundwork for our study of model alignment. We've moved from the "how" of model adaptation to the "why," exploring the critical need to ensure AI systems behave in line with human intentions.
Here are the key takeaways:
- AI Alignment is the challenge of ensuring an AI's goals and behavior are consistent with human values, addressing the gap between the specified objective and the true desired outcome.
- A common practical framework for LLM alignment is the Triple H: making models Helpful, Honest, and Harmless.
- Misalignment manifests in a wide range of failure modes, including fabrication (hallucination), sycophancy, reward hacking, and strategic deception.
- The alignment problem can be formally divided into Outer Alignment (specifying the right goals) and Inner Alignment (ensuring the model truly adopts those goals). Failures like reward hacking are typically considered inner alignment problems.
You now have a vocabulary and a conceptual map for the problems that alignment techniques are designed to solve. In our next lesson, we will begin to explore the most influential of these techniques: Reinforcement Learning from Human Feedback (RLHF). We will dissect its three-stage process to see how it attempts to steer models toward the "Triple H" and away from the failure modes we've discussed today.