Least Squares Cost Function for Linear Regression

Recommended resources

The freely available resources this lesson will direct you to.

Welcome back! In our last lesson, we formulated the linear regression model $y = X β + ϵ$ and detailed the critical statistical assumptions that underpin it. We now have a theoretical model, but a crucial question remains: given a dataset of $(x_{i}, y_{i})$ pairs, how do we choose the "best" possible values for our parameter vector $β$ ?

This lesson tackles that question head-on. Your learning outcome is to derive the cost function for linear regression from a least squares perspective. We will establish a clear, quantifiable measure of how well our model's predictions match the actual data. This measure, the cost function, transforms the task of "fitting a model" into a formal optimization problem, paving the way for us to find the optimal $β$ in subsequent lessons.

1. The Core Idea: Minimizing Prediction Error

For any chosen parameter vector $β$ , our linear model makes a prediction, $y^$ , for a given set of features, $x$ . This prediction is given by the deterministic part of our model:

y^= β_{0} + β_{1} x_{1} + \dots + β_{k} x_{k} = x^{T} β

The goal is to select $β$ such that the predictions $y^_{i}$ are as "close" as possible to the true observed values $y_{i}$ for all the data points in our training set. The difference between the true value and the predicted value is called the residual (or error) for that observation:

e_{i} = y_{i} - y^_{i} = y_{i} - x_{i}^{T} β

Least-Squares Linear Regression Visualization — This diagram illustrates the concept of residuals. The blue dots are the actual data points. The red line is the regression model $\hat{y} = \alpha + \beta x$ (using $\alpha, \beta$ for intercept and slope). The vertical dashed line, $\epsilon_i$, represents the residual for one data point—the difference between the observed value and the value predicted by the line.

Our task is to find the line (i.e., the parameters $β$ ) that makes all these residuals, collectively, as small as possible. This is the principle of "least squares."

2. How to Aggregate Errors: From a Flawed Idea to a Robust Solution

How should we combine all the individual residuals $e_{i}$ into a single number that represents the total error of our model? Let's explore a few possibilities to understand why the "least squares" approach is so widely used.

The following video provides an excellent and intuitive step-by-step derivation of a good error measure. It starts by showing why simpler ideas are flawed.

Mathematical Derivation for Simple Linear Regression Using Least Squares

This video from Data Scholar will walk us through the reasoning for choosing a specific error function. Pay close attention to the arguments against simpler methods.

Please watch the following segments: 08:48 - 14:14: This part demonstrates why simply summing the residuals is a bad idea, as positive and negative errors cancel each other out. 20:01 - 29:13: This section introduces the sum of absolute errors as a potential fix but then explains its critical drawback for optimization: non-differentiability. Your calculus background will be very useful for appreciating this point.

As the video explains, we need an error measure that avoids two key problems:

Cancellation of Errors: Simply summing the residuals, $\sum e_{i}$ , is not viable because large positive and negative residuals can cancel out, making a poorly fitting model appear to be a good one.
Non-Differentiability: Taking the sum of absolute errors, $\sum ∣ e_{i} ∣$ , solves the cancellation problem. However, the absolute value function $∣ x ∣$ has a sharp point at $x = 0$ , where its derivative is undefined. This lack of a smooth derivative everywhere makes it significantly more difficult to find the minimum of the error function using the powerful tools of calculus, which you've studied in the optimization module.

This leads us to the most common and effective solution: squaring the residuals.

3. The Least Squares Cost Function: Sum of Squared Errors

By squaring each residual, we achieve two things simultaneously:

All terms $(e_{i})^{2}$ become non-negative, so there is no risk of cancellation.
The function $f (x) = x^{2}$ is a smooth, convex, and perfectly differentiable parabola. This mathematical property is incredibly convenient, as it guarantees that our resulting cost function will have a unique global minimum that can be found by setting its derivative to zero.

This brings us to the Sum of Squared Errors (SSE), which is the foundation of the least-squares cost function.

Mathematical Derivation for Simple Linear Regression Using Least Squares

Now let's watch the segment that introduces the sum of squared errors and explains its desirable properties.

Please watch these segments from the same 'Data Scholar' video: 29:13 - 32:18: This part formally introduces the Sum of Squared Errors (SSE) as the solution. 32:35 - 43:20: This is a key section. It explains why the SSE function is a 'convex function', ensuring it has a single global minimum. This connects directly to the optimization theory we covered earlier.

Now, let's formalize this with the standard notation used in machine learning. The goal is to find the parameters $β$ that minimize the sum of these squared residuals.

Stanford CS229: Machine Learning - Linear Regression and Gradient Descent | Lecture 2 (Autumn 2018)

This clip from Andrew Ng's Stanford lecture provides the canonical definition of the cost function for linear regression, often denoted J( heta) or, in our notation, J(oldsymbol{eta}).

Watch the segment from 14:17 to 17:52. Andrew Ng defines the cost function, explains the summation over all training examples, and justifies the conventional rac{1}{2} factor as a mathematical convenience for the calculus we'll be doing next.

As you have seen, we can now formally define the cost function.

The Sum of Squared Errors (SSE) is:

SSE (β) = i = 1 \sum n (y_{i} - y^_{i})^{2} = i = 1 \sum n (y_{i} - x_{i}^{T} β)^{2}

Our objective is to find the $β$ that minimizes this function.

β^_{O L S} = ar g β min i = 1 \sum n (y_{i} - x_{i}^{T} β)^{2}

For mathematical convenience in differentiation (which we will perform in the next lesson), the cost function, $J (β)$ , is typically defined with a $\frac{1}{2}$ scaling factor. Minimizing $SSE (β)$ is equivalent to minimizing $\frac{1}{2} SSE (β)$ , as this does not change the location of the minimum.

Thus, the least-squares cost function for linear regression is:

J (β) = \frac{1}{2} i = 1 \sum n (y_{i} - x_{i}^{T} β)^{2}

To solidify this definition, let's look at it one more time in a formal text.

CS229 Lecture notes

This excerpt from the Stanford CS229 lecture notes provides a crisp, formal definition of the least-squares cost function. It's a great reference to have.

Read the first two paragraphs of the document, up to the definition of the cost function J(θ). Note that it uses θ for parameters and m for the number of examples, but the mathematical form is identical to what we've derived.

Test your understanding!

Consider two models. Model A has residuals of $[- 1, 1, - 1, 1]$ . Model B has residuals of $[2, 0, 0, 0]$ .

Calculate the Sum of Residuals for both models. Based on this (flawed) metric, which model appears better?
Calculate the Sum of Absolute Errors for both models.
Calculate the Sum of Squared Errors (SSE) for both models.
Based on SSE, which model is preferred by the least-squares criterion? Why does this make intuitive sense?

Show answer

Sum of Residuals:
- Model A: $- 1 + 1 - 1 + 1 = 0$
- Model B: $2 + 0 + 0 + 0 = 2$
- Based on this metric, Model A appears perfect, which is clearly wrong.
Sum of Absolute Errors:
- Model A: $∣ - 1∣ + ∣1∣ + ∣ - 1∣ + ∣1∣ = 4$
- Model B: $∣2∣ + ∣0∣ + ∣0∣ + ∣0∣ = 2$
Sum of Squared Errors (SSE):
- Model A: $(- 1)^{2} + 1^{2} + (- 1)^{2} + 1^{2} = 1 + 1 + 1 + 1 = 4$
- Model B: $2^{2} + 0^{2} + 0^{2} + 0^{2} = 4$
Based on SSE, both models are considered equally "bad" ( $SSE = 4$ ). The least-squares criterion penalizes larger errors disproportionately. Model A makes several small errors, while Model B makes one large error. In this specific case, the squared penalty on Model B's single large error ( $2^{2} = 4$ ) equals the sum of squared penalties on Model A's four smaller errors ( $1^{2} + 1^{2} + 1^{2} + 1^{2} = 4$ ). This illustrates how SSE is sensitive to outliers or large prediction mistakes.

Conclusion

In this lesson, you have rigorously derived the cost function for linear regression. We started with the intuitive goal of minimizing prediction error and systematically arrived at a mathematically sound and computationally convenient objective: minimizing the sum of squared errors.

Key Takeaways:

The goal of fitting a linear regression model is to find the parameters $β$ that minimize the difference between predicted ( $y^_{i}$ ) and actual ( $y_{i}$ ) values.
Simply summing residuals is flawed due to the cancellation of positive and negative errors.
Summing absolute errors solves the cancellation issue but introduces a non-differentiable cost function, which complicates optimization.
The Sum of Squared Errors (SSE) provides a cost function that is always non-negative, penalizes larger errors more, and is smooth and convex, guaranteeing a unique, findable global minimum.
The formal least-squares cost function is written as $J (β) = \frac{1}{2} \sum_{i = 1}^{n} (y_{i} - x_{i}^{T} β)^{2}$ , where the $\frac{1}{2}$ is a convenience for differentiation.

Preview of the Next Lesson:
We have successfully framed our problem. We have a model, $y^_{i} = x_{i}^{T} β$ , and a cost function, $J (β)$ , that tells us how good any choice of $β$ is. The next logical step is to find the $β$ that minimizes this cost. In the next lesson, you will apply your skills from multivariate calculus to derive the closed-form solution for the Ordinary Least Squares (OLS) estimator by calculating the gradient of $J (β)$ , setting it to zero, and solving for $β$ .