Loss Functions: Regression vs. Classification

Quantifying Error in Regression

Let's begin in familiar territory. In regression, the goal is to predict a continuous numerical value—like the price of a house or the temperature tomorrow. As we saw in the last lesson, a common way to measure the error for these tasks is the Mean Squared Error (MSE), also known as L2 Loss.

For a dataset with $n$ examples, the MSE is calculated as the average of the squared differences between the true values ($y_i$) and the model's predictions (${\hat{y}}_i$).

The key characteristics of MSE are:

• Quadratic Penalty: By squaring the error, it penalizes larger mistakes much more heavily than smaller ones. A prediction that is off by 2 units contributes 4 to the loss, while a prediction off by 4 units contributes 16.
• Sensitivity to Outliers: This quadratic penalty also makes the model highly sensitive to outliers. A single data point with a very large error can dominate the loss value and pull the model's predictions strongly towards it.
• Smoothness: The MSE function is a smooth, convex parabola, which means it has a single global minimum and is easy to optimize with gradient-based methods.

An alternative is the Mean Absolute Error (MAE), or L1 Loss, which averages the absolute differences:

MAE is less sensitive to outliers because the penalty scales linearly, not quadratically. However, its derivative is not continuous at zero, which can add complexity to the optimization process.

The following video provides an excellent visual comparison of these regression losses and their behavior, particularly in the presence of outliers. It also introduces a hybrid, the Pseudo-Huber loss, which combines the best properties of both.

Quantifying Error in Classification

Now, let's shift to classification, where the goal is to predict a discrete class label (e.g., "spam" or "not spam", "cat" or "dog"). Here, the model's output is typically a probability distribution over the possible classes. For a binary problem, the model might output a single number, say 0.9, representing a 90% probability that the sample belongs to the positive class.

Could we use MSE for this? While technically possible, it's not ideal for several reasons. Most importantly, MSE doesn't strongly penalize a model that is confidently wrong. For a true label of 1 (positive class), MSE treats a prediction of 0.1 (confidently wrong) not much differently than a prediction of 0.4 (uncertain). We need a loss function that punishes confident errors severely.

This is where Cross-Entropy Loss comes in. It originates from information theory and measures the "distance" or "divergence" between two probability distributions: the true distribution and the one predicted by the model.

The general formula for cross-entropy between a true distribution $P$ and a predicted distribution $Q$ over $C$ classes is:

In machine learning, $P$ is the ground truth (e.g., a one-hot encoded vector like [0, 1, 0]), and $Q$ is the vector of probabilities predicted by our model (e.g., [0.1, 0.8, 0.1]).

The logarithmic term is the key. If the model assigns a very low probability to the correct class, $\log (Q(i))$ will be a large negative number, resulting in a very high loss. This provides a steep gradient that allows the model to learn quickly from its confident mistakes.

Let's explore the two main variants of cross-entropy.

To summarize the key points from the reading:

• Binary Cross-Entropy (BCE), or Log Loss, is used for binary (two-class) classification. Its formula for a single prediction is $-[y\log (\hat{y})+(1-y)\log (1-\hat{y})]$, where $y$ is the true label (0 or 1) and $\hat{y}$ is the predicted probability for the positive class. It's typically paired with a sigmoid activation function.
• Categorical Cross-Entropy (CCE) is used for multi-class classification where classes are mutually exclusive. Since the true label is one-hot encoded (e.g., [0, 1, 0]), the formula simplifies to just the negative logarithm of the probability predicted for the correct class: $-\log ({\hat{y}}_c)$. It's paired with a softmax activation function, which ensures the output probabilities sum to 1.

Visualizing the Difference: MSE vs. Cross-Entropy

The most compelling reason to prefer cross-entropy for classification is how it behaves during training. Let's visualize the loss functions and their gradients.

The following video provides an animated walkthrough of this very scenario, demonstrating how the much larger gradient from cross-entropy loss leads to more significant and effective model updates when it makes a confident mistake.

The Unifying Principle: Maximum Likelihood Estimation

So far, MSE for regression and cross-entropy for classification seem like two distinct tools for different jobs. However, your background in mathematics and computer science provides the perfect foundation to appreciate a deeper, unifying principle that connects them: Maximum Likelihood Estimation (MLE).

In essence, choosing a loss function is equivalent to making an assumption about the probability distribution of your target data. Training a model to minimize a loss function is often equivalent to finding the model parameters that maximize the likelihood of observing your training data under that assumed distribution.

• Minimizing MSE is equivalent to maximizing the log-likelihood of the data, assuming the target values $y$ are drawn from a Gaussian (Normal) distribution centered around your model's prediction $\hat{y}$.
• Minimizing Cross-Entropy is equivalent to maximizing the log-likelihood of the data, assuming the target labels $y$ are drawn from a Bernoulli distribution (for BCE) or a Categorical distribution (for CCE).

This is a profound insight. It reframes the choice of a loss function from a simple heuristic to a principled decision about the nature of your data. The following article beautifully derives the connection for MSE, showing how it emerges directly from the log-likelihood of a Gaussian distribution.

Conclusion

In this lesson, we moved from measuring error to understanding how to formulate it as a loss function that can be optimized. This is the heart of model training.

Here are the key takeaways:

• Loss functions quantify the difference between a model's predictions and the true values. The goal of training is to minimize this loss.
• Mean Squared Error (MSE) is the standard for regression. It penalizes errors quadratically, making it sensitive to outliers but easy to optimize.
• Cross-Entropy Loss is the standard for classification. It measures the divergence between the predicted and true probability distributions.
• The primary advantage of cross-entropy for classification is its large gradient for confident misclassifications, which drives faster and more effective learning compared to MSE.
• From a probabilistic perspective (MLE), minimizing MSE assumes a Gaussian distribution for the targets, while minimizing cross-entropy assumes a Bernoulli/Categorical distribution.

Now that we know how to define and measure the error for a given set of model parameters, the next logical question is: how do we systematically find the parameters that minimize this error? In our next lesson, we will answer this by implementing logistic regression for binary classification and deriving the gradient descent algorithm to optimize its parameters.