Loss Functions and Maximum Likelihood Estimation

1. The Principle of Maximum Likelihood Estimation (MLE)

At its core, training a supervised machine learning model is a search problem. We are searching for the set of model parameters, $\theta$, that best explains the data we've observed. The principle of Maximum Likelihood Estimation (MLE) formalizes this. It directs us to choose the parameters $\theta$ that maximize the probability—or more accurately, the likelihood—of observing our training dataset.

Given a set of $N$ independent and identically distributed (i.i.d.) data points $(x_i,y_i)$, the likelihood of the entire dataset is the product of the probabilities of each individual data point:

Here, $P(y_i|x_i;\theta )$ is the probability of observing label $y_i$ given input $x_i$, according to our model with parameters $\theta$.

Working with products is computationally difficult and can lead to numerical underflow. It's much easier to work with sums. Since the logarithm is a monotonically increasing function, maximizing the likelihood is equivalent to maximizing the log-likelihood:

Most optimization frameworks are designed to minimize a loss function. Therefore, we can frame our goal as minimizing the negative log-likelihood (NLL):

This simple reframing is the bridge between statistical estimation and the loss functions you encounter in deep learning.

2. Example: Why Mean Squared Error is a Negative Log-Likelihood

Let's make this concrete with a familiar loss function: Mean Squared Error (MSE). You've likely used it for regression tasks, but why is it the standard choice? The answer lies in MLE.

For a regression problem, we often implicitly assume that our target variable $y$ is a combination of our model's deterministic prediction $f_{\theta }(x)$ and some random Gaussian noise $ϵ$, where $ϵ\sim \mathcal{N}(0,{\sigma }^2)$. This is equivalent to assuming that the probability of observing a specific $y$ given $x$ follows a Gaussian (normal) distribution centered at our model's prediction:

The probability density function (PDF) for a single point $(x_i,y_i)$ is therefore:

The negative log-likelihood for a single data point is:

To find the parameters $\theta$ that maximize the likelihood for the entire dataset, we minimize the sum of NLL over all points. When we do this, the first term, $\frac{1}{2}\log (2\pi {\sigma }^2)$, is a constant with respect to $\theta$. The $2{\sigma }^2$ in the denominator is also a constant scaling factor. Therefore, minimizing the NLL is equivalent to minimizing:

This is precisely the Mean Squared Error loss (ignoring the constant factor $\frac{1}{N}$). This derivation shows that using MSE is equivalent to performing maximum likelihood estimation under the assumption of Gaussian noise.

3. Cross-Entropy and KL Divergence: Measuring the Distance Between Distributions

While the Gaussian assumption works for regression, it's unsuitable for classification, where outputs are discrete class labels. In classification, our model outputs a probability distribution (via softmax), and our goal is to make this distribution as "close" as possible to the true distribution (which is a one-hot vector representing the correct class).

Information theory provides the tools to measure this "closeness":

• Cross-Entropy: Suppose the true data distribution is $p$ and our model's predicted distribution is $q$. The cross-entropy, $H(p,q)$, measures the average number of bits required to encode data from $p$ when using a code optimized for $q$. The formula is: $$H(p,q)=-\sum _i{p}_i\log q_i$$
• Kullback-Leibler (KL) Divergence: This measures the extra bits required because we used the "wrong" distribution $q$ instead of the true one $p$. It's a measure of how much $q$ diverges from $p$. $$D_{\text{KL}}(p\Vert q)=H(p,q)-H(p)=\sum _i{p}_i\log \frac{p_i}{q_i}$$ where $H(p)$ is the entropy of the true distribution. Since $H(p)$ is a constant with respect to our model's parameters, minimizing the KL divergence is equivalent to minimizing the cross-entropy.

Training a classification model is therefore an exercise in minimizing the KL divergence between the true data distribution $p$ and our model's predicted distribution $q$.

4. The Unifying Trinity: MLE, KL Divergence, and Cross-Entropy

We have now established two key equivalences:

1. Minimizing Negative Log-Likelihood $⟺$ Maximum Likelihood Estimation.
2. Minimizing Cross-Entropy $⟺$ Minimizing KL Divergence.

The final step is to connect these two worlds. It turns out that performing Maximum Likelihood Estimation is mathematically identical to minimizing the KL divergence between the empirical distribution of your data and your model's distribution.

Let ${\hat{p}}_{\text{data}}$ be the empirical distribution from our training data (i.e., the distribution where the probability of a data point is $1/N$ if it's in our dataset, and 0 otherwise). Minimizing the KL divergence $D_{\text{KL}}({\hat{p}}_{\text{data}}\Vert p_{\theta })$ can be shown to be equivalent to minimizing the negative log-likelihood of the data under the model $p_{\theta }$.

This reveals the profound unity of these three concepts:

• Maximum Likelihood Estimation (Statistics): Choose parameters that make the data most probable.
• KL Divergence Minimization (Information Theory): Make the model distribution as close as possible to the data distribution.
• Cross-Entropy Minimization (Machine Learning): Minimize the cross-entropy loss function.

These are three different perspectives on the exact same optimization problem.

To reinforce this key idea, the following video offers a concise, practical explanation.

5. Practical Implications for Model Training

This theoretical foundation has direct, practical consequences for engineering robust AI systems.

Why Cross-Entropy is Superior to MSE for Classification

The primary reason is its effect on gradients during backpropagation. When you combine a softmax output with a cross-entropy loss, the gradient of the loss with respect to the pre-softmax activations (the logits, $z_j$) has a beautifully simple form:

where ${\hat{y}}_j$ is the predicted probability and $y_j$ is the true label (0 or 1). The gradient is simply the error in the prediction. When the model is confidently wrong (e.g., ${\hat{y}}_j$ is near 0 but $y_j$ is 1), the gradient is large, providing a strong learning signal.

In contrast, the gradient for MSE loss with a sigmoid/softmax output contains a ${\sigma }^{\prime }(z)$ term. This derivative term approaches zero when the activation saturates (i.e., when the output is near 0 or 1). This means if the model is confidently wrong, the gradients vanish, and learning stalls. Cross-entropy's probabilistic formulation elegantly sidesteps this critical problem.

The Asymmetry of KL Divergence

A final, advanced point is that KL divergence is not a true "distance" metric because it is asymmetric: $D_{\text{KL}}(p\Vert q)\ne D_{\text{KL}}(q\Vert p)$. The direction matters.

• Forward KL $D_{\text{KL}}(p\Vert q)$, used in MLE, is "mean-seeking." It tries to ensure that where $p$ has probability mass, $q$ also has probability mass. It leads to inclusive, broader approximations.
• Reverse KL $D_{\text{KL}}(q\Vert p)$ is "mode-seeking." It tries to ensure that where $q$ has high probability mass, $p$ also does. It punishes $q$ for putting mass in places where $p$ has none, leading to exclusive, sharper approximations that focus on a single mode.

While standard classification uses the forward KL implicitly through MLE, understanding this distinction is crucial for more advanced generative models like Variational Autoencoders (VAEs), which often use the reverse form.

Conclusion

In this lesson, you have completed your mathematical toolkit by connecting the worlds of probability, information theory, and machine learning. You've moved beyond simply using loss functions to understanding their fundamental justification.

Key takeaways from this lesson include:

• Common loss functions are not arbitrary; they are derived from the principle of Maximum Likelihood Estimation (MLE).
• Minimizing the Negative Log-Likelihood (NLL) is the operational way to perform MLE. For regression with a Gaussian noise assumption, this yields Mean Squared Error.
• For classification, training a model is equivalent to minimizing the KL Divergence between the data distribution and the model's predicted distribution.
• Minimizing KL Divergence is, in turn, equivalent to minimizing the Cross-Entropy loss.
• This establishes the core equivalence: MLE $⟺$ KL Divergence Minimization $⟺$ Cross-Entropy Minimization.
• This probabilistic view explains why Cross-Entropy provides more stable gradients than MSE for classification tasks, preventing the saturation problem.

Having established the what (calculus), the how (linear algebra), and the why (probability), we are now ready to explore how modern frameworks put this all together. In the next lesson, we will investigate the mechanics of automatic differentiation and the structure of dynamic computational graphs, revealing the magic that powers frameworks like PyTorch and enables the efficient computation of the gradients we've discussed.