MAP, MLE, and L2 Regularization: A Relationship Explained

1. From Likelihood to Regularization: A Conceptual Overview

Before diving into the derivation, let's build the high-level intuition. We'll start by revisiting linear regression, the problem of overfitting, and how regularization is introduced as a solution. Then, we'll see how a Bayesian viewpoint provides a principled origin for this technique.

The video sets the stage perfectly. We have two seemingly different approaches to fitting a linear model:

1. Regularized Least Squares: Minimize the sum of squared errors plus a penalty term on the size of the coefficients ($\beta$).
2. MAP Estimation: Maximize the posterior probability, which is proportional to the likelihood of the data times a prior probability on the coefficients.

Our goal now is to prove that these two approaches are, in fact, mathematically identical.

2. The Mathematical Equivalence of MAP and L2 Regularization

Let's begin the formal derivation. Your background in mechanical engineering involves modeling physical systems with equations; here, we are doing the same for a statistical system. We will build the model piece by piece, starting with the likelihood (the "physics" of the data-generating process) and then adding the prior (our "engineering assumptions" about the parameters).

Step 1: The Likelihood (Connection to MLE and Least Squares)

We begin with the standard linear model:
$\mathbf{y}=\mathbf{X}\beta +ϵ$
where $\mathbf{y}$ is the vector of observations, $\mathbf{X}$ is the design matrix, $\beta$ is the vector of coefficients, and $ϵ$ is the error term. We assume the errors are independent and identically distributed, following a zero-mean Gaussian distribution: $ϵ\sim \mathcal{N}(0,{\sigma }^{2}I)$.

This assumption implies that the likelihood of observing $\mathbf{y}$ given the parameters $\beta$ is also Gaussian:
$p(\mathbf{y}|\mathbf{X},\beta ,{\sigma }^2)\sim \mathcal{N}(\mathbf{X}\beta ,{\sigma }^{2}I)$
The probability density function is:
$p(\mathbf{y}|\beta )=\frac{1}{(2\pi {\sigma }^2{)}^{N/2}}\exp \left(-\frac{1}{2{\sigma }^2}||\mathbf{y}-\mathbf{X}\beta |{|}_2^2\right)$
To find the Maximum Likelihood Estimate (MLE), we maximize this function (or its log) with respect to $\beta$.
${\hat{\beta }}_{\text{MLE}}=\arg {\max }_{\beta }\left[\log p(\mathbf{y}|\beta )\right]$
${\hat{\beta }}_{\text{MLE}}=\arg {\max }_{\beta }\left[-\frac{N}{2}\log (2\pi {\sigma }^2)-\frac{1}{2{\sigma }^2}||\mathbf{y}-\mathbf{X}\beta |{|}_2^2\right]$
Dropping the constant terms and the negative sign (which turns the argmax into an argmin), we get:
${\hat{\beta }}_{\text{MLE}}=\arg {\min }_{\beta }||\mathbf{y}-\mathbf{X}\beta |{|}_2^2$
This is exactly the Ordinary Least Squares (OLS) objective function. Thus, OLS is equivalent to finding the MLE for a linear model with Gaussian noise.

The following resource provides a detailed derivation of this fact in the context of polynomial regression.

Step 2: Adding the Prior

Now, let's move from MLE to MAP by introducing a prior on the weights $\beta$. A common and sensible belief is that simpler models are better, which often translates to a preference for smaller coefficient values. A mathematically convenient way to express this is with a zero-mean Gaussian prior on each coefficient:
${\beta }_j\sim \mathcal{N}(0,{\tau }^2)$
Assuming the coefficients are independent, the prior for the entire vector $\beta$ is:
$p(\beta )={\prod }_{j=1}^P\frac{1}{\sqrt{2\pi {\tau }^2}}\exp \left(-\frac{{\beta }_j^2}{2{\tau }^2}\right)=\frac{1}{(2\pi {\tau }^2{)}^{P/2}}\exp \left(-\frac{1}{2{\tau }^2}||\beta |{|}_2^2\right)$
Here, ${\tau }^2$ is a hyperparameter that controls the width of our prior. A small ${\tau }^2$ means we have a strong prior belief that the weights should be very close to zero. A large ${\tau }^2$ represents a weak, or "uninformative," prior.

Step 3: Deriving the MAP Estimate

The MAP estimate maximizes the log-posterior, which is the sum of the log-likelihood and the log-prior:
${\hat{\beta }}_{\text{MAP}}=\arg {\max }_{\beta }\left[\log p(\mathbf{y}|\beta )+\log p(\beta )\right]$
Substituting the expressions we found:
${\hat{\beta }}_{\text{MAP}}=\arg {\max }_{\beta }\left[\left(C_1-\frac{1}{2{\sigma }^2}||\mathbf{y}-\mathbf{X}\beta |{|}_2^2\right)+\left(C_2-\frac{1}{2{\tau }^2}||\beta |{|}_2^2\right)\right]$
where $C_1$ and $C_2$ are constants with respect to $\beta$. Dropping these constants and flipping the argmax to an argmin by negating the expression, we get:
${\hat{\beta }}_{\text{MAP}}=\arg {\min }_{\beta }\left[\frac{1}{2{\sigma }^2}||\mathbf{y}-\mathbf{X}\beta |{|}_2^2+\frac{1}{2{\tau }^2}||\beta |{|}_2^2\right]$
We can multiply this objective by a positive constant, $2{\sigma }^2$, without changing the location of the minimum:
${\hat{\beta }}_{\text{MAP}}=\arg {\min }_{\beta }\left[||\mathbf{y}-\mathbf{X}\beta |{|}_2^2+\frac{{\sigma }^2}{{\tau }^2}||\beta |{|}_2^2\right]$

Step 4: The Final Connection

Look closely at this final expression. It is precisely the objective function for Ridge Regression (L2 Regularization):
${\hat{\beta }}_{\text{Ridge}}=\arg {\min }_{\beta }\left[||\mathbf{y}-\mathbf{X}\beta |{|}_2^2+\lambda ||\beta |{|}_2^2\right]$
We have just proven that they are the same problem, where the regularization parameter $\lambda$ is given by the ratio of the variances:
$\lambda =\frac{{\sigma }^2}{{\tau }^2}=\frac{\text{variance of data noise}}{\text{variance of prior belief}}$

This entire derivation is laid out very clearly in the following resources.

3. Visualizing and Interpreting the Relationship

The equivalence $\lambda ={\sigma }^2/{\tau }^2$ provides a profound intuition about regularization.

• Strong Prior (small ${\tau }^2$): If we are very confident that the weights should be small, our prior distribution is narrow. This leads to a large $\lambda$, resulting in strong regularization. The model is forced to keep its weights small, trusting our prior belief more than the data.
• Weak Prior (large ${\tau }^2$): If we are uncertain about the weights, our prior is wide and flat. This leads to a small $\lambda$, resulting in weak regularization. The model is free to fit the data, as in OLS. In the limit as ${\tau }^2\to \infty$, the prior becomes uniform, $\lambda \to 0$, and MAP converges to MLE.

The diagram below visualizes how the MAP estimate is a balance between the data (likelihood) and the prior belief.

The choice of a Gaussian prior leads to L2 regularization. Other priors lead to other forms of regularization. For instance, a Laplace (or double-exponential) prior, which is more sharply peaked at zero, leads to L1 regularization (Lasso), which famously encourages some weights to be exactly zero.

Conclusion

Today, we have forged a crucial link between Bayesian inference and a cornerstone of practical machine learning. By working through the derivation, we have shown that what might appear as a mathematical convenience—adding a penalty term to a loss function—has a deep probabilistic justification.

Key Takeaways:

• MLE and Least Squares: For a linear model with Gaussian noise, the Maximum Likelihood Estimate is identical to the Ordinary Least Squares solution.
• MAP and L2 Regularization: MAP estimation of a linear model's parameters, using a zero-mean Gaussian prior on those parameters, is mathematically equivalent to L2-regularized least squares (Ridge Regression).
• The Regularization Parameter $\lambda$: The hyperparameter $\lambda$ is not arbitrary; it represents the ratio of the data noise variance (${\sigma }^2$) to the prior belief variance (${\tau }^2$). It quantifies the balance between trusting the data and trusting our prior belief in a simpler model.
• MLE as a special case of MAP: When the prior is uninformative (e.g., a uniform distribution, or a Gaussian with infinite variance), MAP estimation reduces to MLE. This corresponds to setting the regularization strength $\lambda$ to zero.

Preview of the Next Lesson:

Having explored the theoretical underpinnings of MMSE and MAP estimation, it's time to put these ideas into practice. In the next lesson, you will undertake a project: implementing a Bayesian estimator to sequentially update your belief about a coin's bias as new data arrives. This will allow you to see firsthand how a prior distribution is transformed into a posterior distribution with each new observation, solidifying the concepts from this module.