Loss Functions and Gradient Descent in Model Training

Welcome back! In our previous lesson, we established the fundamental goal of machine learning: building models that generalize well to new data by finding a sweet spot in the bias-variance tradeoff. But this raises a crucial question: how does a model actually learn from data to find this optimal balance?

Today, we'll dive into the mechanics of the training process. You will learn to describe the two core components that enable a model to learn:

Loss Functions: These act as a "scorecard," quantifying how wrong a model's predictions are. We'll focus on Mean Squared Error (MSE) for regression and Cross-Entropy for classification.
Gradient Descent: This is the optimization algorithm that works like a GPS, guiding the model to iteratively adjust its parameters (weights and bias) to minimize the loss and improve its score.

This process is essentially an optimization problem, a concept that may be familiar from your computational mathematics and algorithms coursework. We're searching for the best set of model parameters in a vast landscape of possibilities.

Measuring Error: The Role of the Loss Function

Before we can improve a model, we need a way to measure its performance. A loss function (also called a cost function or objective function) does exactly this. It takes the model's predictions and the true target values and computes a single number representing the total error. The entire goal of training is to find the model parameters that make this number as small as possible.

Let's look at the two most common loss functions.

1. Mean Squared Error (MSE) for Regression

When predicting a continuous value, like a house price or a temperature, Mean Squared Error is the standard loss function. It calculates the average of the squared differences between the predicted values and the actual values.

The formula is:

MSE = \frac{1}{n} i = 1 \sum n (y_{i} - y^_{i})^{2}

Where:

$n$ is the number of data points.
$y_{i}$ is the true value for the $i$ -th data point.
$y^_{i}$ is the model's predicted value for the $i$ -th data point.

The key intuition here is the squaring of the error term $(y_{i} - y^_{i})$ . This means that larger errors are penalized much more heavily than smaller ones. For example, an error of 4 is 16 times worse than an error of 1.

2. Cross-Entropy for Classification

When predicting a class or a probability (e.g., "spam" or "not spam"), Cross-Entropy (or Log Loss) is the preferred choice. Intuitively, it measures how different the model's predicted probability distribution is from the true distribution. A perfect model that assigns a probability of 1.0 to the correct class would have a cross-entropy loss of 0.

For a binary classification problem, the formula is:

Cross-Entropy = - \frac{1}{n} i = 1 \sum n [y_{i} lo g (p^_{i}) + (1 - y_{i}) lo g (1 - p^_{i})]

Where:

$y_{i}$ is the true label (0 or 1).
$p^_{i}$ is the model's predicted probability that the class is 1.

This formula cleverly penalizes confident but incorrect predictions. If the true label is 1 ( $y_{i} = 1$ ) but the model predicts a probability close to 0 ( $p^_{i} \approx 0$ ), the term $lo g (p^_{i})$ becomes a very large negative number, resulting in a huge loss.

The image below provides a great summary of these and other common loss functions.

A handy reference for common loss functions used in machine learning, showing their formulas, what they penalize, and their typical applications. Source: Miro/Medium.

Minimizing Error: The Gradient Descent Algorithm

Now that we can score our model with a loss function, how do we find the weights and biases that give us the lowest score? This is where Gradient Descent comes in.

Imagine you are standing on a foggy hillside (the loss surface) and want to get to the lowest point (the minimum loss). The best you can do is look at the ground at your feet to see which direction is steepest downwards and take a step in that direction. You repeat this process until you can no longer step down. This is the core intuition of Gradient Descent.

Linear regression: Gradient descent | Machine Learning

This article from Google's Machine Learning Crash Course gives a great high-level overview of the iterative nature of Gradient Descent.

Read the following sections: Page Summary: This provides a concise overview of the entire process. The main text and diagram right below "Figure 11": Focus on the three iterative steps: calculate loss, determine direction, and update parameters. "Model convergence and loss curves": This section introduces the "loss curve," a crucial tool for visualizing how a model's error decreases over training iterations. Pay attention to how the curve flattens out as the model converges.

The "direction of steepest descent" is found by calculating the gradient of the loss function. The gradient is a vector of partial derivatives, where each element tells us how the loss would change if we slightly nudged one of the model's parameters. We then update each parameter by taking a small step in the direction opposite to the gradient.

The size of that step is controlled by a hyperparameter called the learning rate.

A Step-by-Step Walkthrough of Gradient Descent

Let's make this concrete. We'll see exactly how Gradient Descent finds the best intercept and slope for a simple linear regression model by minimizing the MSE loss.

Gradient Descent, Step-by-Step

This StatQuest video, "Gradient Descent, Step-by-Step," provides one of the most intuitive explanations of this algorithm.

Watch the following segments carefully: The Loss Function: See how the sum of squared residuals (our MSE loss) is calculated for a given line. The Iterative Process: This is the core of the lesson. Follow along as the video: Starts with a random parameter (intercept = 0). Calculates the derivative (gradient) of the loss function with respect to that parameter. Calculates the step size (learning rate × gradient). Updates the parameter (new_intercept = old_intercept - step_size). Repeats this process, showing how each step gets smaller as it approaches the minimum loss. Optimizing Two Parameters: This part extends the concept to optimizing both the slope and intercept simultaneously. It explains that when you have derivatives for multiple parameters, the collection of these derivatives is called the gradient. General Steps: This is a fantastic summary of the entire Gradient Descent algorithm, applicable to any loss function.

Unifying the Concepts: Loss Functions and Gradient Descent in Practice

The framework of a loss function + Gradient Descent is not limited to linear regression. It's a general recipe for training a wide variety of machine learning models. Let's see how it applies to both regression and classification.

Gradient Descent Algorithm in Machine Learning

This GeeksforGeeks article provides Python code examples that bring these concepts to life. Your background in Python will make it easy to follow.

Focus on these two sections: "1) Linear Regression": This section recaps what we just saw in the video, but this time with code. Notice how the w (weight) and b (bias) are updated in a loop using the calculated gradients dw and db to minimize the MSE loss. "2) Logistic Regression (Binary Classification)": Here, the model and loss function are different, but the optimization process is the same. The model uses a sigmoid function to predict probabilities, the Cross-Entropy loss is used to measure error, and Gradient Descent is once again used to update w and b to minimize that loss.

This demonstrates the power and flexibility of this framework. By swapping out the loss function, we can adapt the same learning algorithm to solve entirely different types of problems.

Why Cross-Entropy is Better for Classification

You might wonder why we can't just use MSE for classification. While technically possible, Cross-Entropy is far more effective. The reason lies in the shape of the loss function and the size of the gradients it produces.

Neural Networks Part 6: Cross Entropy

This StatQuest video on Cross-Entropy provides the intuition.

Watch these parts to understand why Cross-Entropy is the right tool for the job: Calculating Cross-Entropy: This segment shows how simple the calculation is for a given prediction. The key is that we only care about the predicted probability for the correct class. Cross-Entropy vs. Squared Residuals: This is the crucial part. It shows that when a model makes a really bad prediction (e.g., predicts 0.1 when the true answer is 1), the Cross-Entropy loss "explodes," creating a very large gradient. This large gradient results in a large update step, helping the model learn much faster from its big mistakes compared to MSE.

Conclusion

In this lesson, you've unpacked the engine room of model training. We've seen that learning is not magic, but a methodical, iterative process of optimization.

Key Takeaways:

Model training is the process of minimizing a loss function.
Mean Squared Error (MSE) is the standard loss function for regression tasks, heavily penalizing large errors.
Cross-Entropy is the standard for classification, penalizing confident wrong predictions and enabling faster learning.
Gradient Descent is the algorithm that minimizes the loss. It iteratively calculates the gradient (direction of steepest ascent) and takes a step in the opposite direction.
The new_parameter = old_parameter - learning_rate * gradient update rule is the heart of the algorithm.

We now understand how to find the parameters that best fit our training data. However, as we learned in the previous lesson, a perfect fit on the training data can lead to overfitting. In our next lesson, we will explore regularization, a powerful technique that modifies the loss function itself to explicitly combat overfitting and improve our model's ability to generalize.

Recommended resources