Understanding the Linear Regression Model

Hello! Welcome to the first lesson of our new module, where we transition from the abstract mathematical foundations to our first concrete machine learning model: Linear Regression. In the previous modules, you've built a robust toolkit of concepts from linear algebra, calculus, optimization, and statistics. Now, we'll start putting them to work.

Your learning outcome for this lesson is to formulate the linear regression model, specifying its components and statistical assumptions. We will define what the model is, identify its parts using precise mathematical language, and, most importantly, understand the "rules of the game" — the key assumptions that must hold for the model to be reliable.

1. From Data to a Model: Defining the Linear Hypothesis

Linear regression is a cornerstone of supervised learning. The goal is to predict a continuous target value, $y$, based on a set of input features, $\mathbf{x}$. Your previous project on house price prediction is a classic example of a regression problem.

Let's begin by formalizing the structure of the model. We'll watch a few short clips from a lecture by Andrew Ng, which clearly introduces the core components and notation.

As you saw, we can define the model for a single data point $i$ with $k$ features as a linear equation. We'll adopt the notation common in statistics and econometrics, using $\beta$ for the parameters and $ϵ$ for the error term.

The model for a single observation $i$ is:

$$y_i={\beta }_0+{\beta }_1{x}_{i1}+{\beta }_2{x}_{i2}+\cdots +{\beta }_k{x}_{ik}+{ϵ}_i$$

Let's break down the components:

• $y_i$: The dependent variable (or target) for observation $i$. This is the value we want to predict (e.g., the price of house $i$).
• $x_{ij}$: The value of the $j$-th independent variable (or feature) for observation $i$ (e.g., the size or number of bedrooms of house $i$).
• ${\beta }_j$: The $j$-th parameter (or coefficient). ${\beta }_j$ represents the change in $y$ for a one-unit change in $x_j$, holding all other features constant (a concept known as ceteris paribus). ${\beta }_0$ is the intercept, representing the value of $y$ when all features are zero.
• ${ϵ}_i$: The error term (or disturbance) for observation $i$. This is a crucial component. It's a random variable that captures all factors affecting $y_i$ that are not included in our model. If our model were perfect, ${ϵ}_i$ would be zero.

2. The Model in Matrix Form

Writing the equation for every data point is cumbersome. As you know from your linear algebra module, matrix notation provides a more compact and powerful way to represent the system for all $n$ observations in our dataset.

This is a fundamental representation that we will use for all derivations in the upcoming lessons.

To summarize the matrix formulation:

Let:

• $\mathbf{y}$ be an $n\times 1$ vector of dependent variables.
• $\mathbf{X}$ be an $n\times (k+1)$ matrix of independent variables (also called the design matrix), where the first column is all ones to account for the intercept ${\beta }_0$.
• $\mathbf{\beta }$ be a $(k+1)\times 1$ vector of parameters.
• $\mathbf{ϵ}$ be an $n\times 1$ vector of error terms.

The entire linear regression model can be written in one simple equation:

$$\mathbf{y}=\mathbf{X}\mathbf{\beta }+\mathbf{ϵ}$$

This compact form is not just for neatness; it's essential for deriving the solution for $\mathbf{\beta }$, which we will do in a future lesson.

3. The Statistical Assumptions of the Model

The equation $\mathbf{y}=\mathbf{X}\mathbf{\beta }+\mathbf{ϵ}$ is just a mathematical statement. To make it a statistical model and to ensure that our estimates for $\mathbf{\beta }$ have desirable properties (like being unbiased), we must make several crucial assumptions about the data generating process, particularly about the error term $\mathbf{ϵ}$. These are known as the Classical Linear Regression Model (CLRM) assumptions.

The following document provides a concise list of these assumptions. We will then explore the most important ones in more detail.

Let's dissect the key assumptions with the mathematical rigor you're looking for.

Assumption 1: Linearity in Parameters

The model must be linear in the parameters $\mathbf{\beta }$. This means the ${\beta }_j$ terms are not squared, exponentiated, or otherwise transformed.

• Why it matters: It ensures the model is a simple weighted sum of the features. Note that the features $X$ themselves can be non-linear (e.g., $x^2$, $\log (x)$). For example, $y={\beta }_0+{\beta }_{1}x+{\beta }_2{x}^2+ϵ$ is still a linear regression model because it's linear in ${\beta }_0,{\beta }_1,{\beta }_2$.

Assumption 2: No Perfect Collinearity

The $n\times (k+1)$ data matrix $\mathbf{X}$ must have full column rank, which is $k+1$.

• Mathematical Form: $\text{rank}(\mathbf{X})=k+1$.
• Why it matters: This means no feature can be written as an exact linear combination of the other features. If you had age_in_years and age_in_months as features, this assumption would be violated. If perfect collinearity exists, the matrix ${\mathbf{X}}^T\mathbf{X}$ is singular and cannot be inverted, which prevents us from finding a unique solution for $\mathbf{\beta }$.

Assumption 3: Zero Conditional Mean (Strict Exogeneity)

This is arguably the most important and conceptually challenging assumption. It states that the expected value of the error term, conditional on the entire data matrix $\mathbf{X}$, is zero.

• Mathematical Form: $E(\mathbf{ϵ}|\mathbf{X})=0$, which implies $E({ϵ}_i|\mathbf{X})=0$ for each observation $i$.
• Why it matters: This assumption implies that the unobserved factors captured by the error term are uncorrelated with the features in our model. If this assumption is violated (a situation called endogeneity), our estimates of $\mathbf{\beta }$ will be biased. A common cause is omitted variable bias, where a relevant variable is left out of the model and is correlated with both the included features and the dependent variable.

The following resource provides a rigorous exploration of this assumption and its profound implications.

Assumption 4: Spherical Errors (Homoscedasticity and No Autocorrelation)

This assumption combines two distinct ideas about the variance of the errors.

• Homoscedasticity (Constant Variance): The variance of the error term is the same for all observations.
  • Mathematical Form: $Var({ϵ}_i|\mathbf{X})={\sigma }^2$ for all $i$.
• No Autocorrelation: The error terms for any two different observations are uncorrelated.
  • Mathematical Form: $Cov({ϵ}_i,{ϵ}_j|\mathbf{X})=0$ for all $i\ne j$.

These two conditions can be summarized beautifully in matrix form: the conditional variance-covariance matrix of the error vector $\mathbf{ϵ}$ is a scaled identity matrix.

• Combined Mathematical Form: $Var(\mathbf{ϵ}|\mathbf{X})={\sigma }^2{\mathbf{I}}_n$.
• Why it matters: While OLS estimators remain unbiased even if this assumption is violated, it is crucial for ensuring that OLS is the most efficient (i.e., has the lowest variance) among all linear unbiased estimators. Violations (heteroscedasticity or autocorrelation) mean our standard errors will be wrong, leading to faulty hypothesis tests.

Assumption 5: Normality of Errors (Optional, for Inference)

For many purposes, we also assume the errors are normally distributed.

• Mathematical Form: $\mathbf{ϵ}|\mathbf{X}\sim \mathcal{N}(0,{\sigma }^2{\mathbf{I}}_n)$.
• Why it matters: This assumption is not required for the unbiasedness or efficiency of the OLS estimator. However, it is essential for conducting statistical inference—that is, performing hypothesis tests (e.g., is ${\beta }_1$ significantly different from zero?) and constructing confidence intervals for our parameters.

This image provides a helpful visual summary of what these assumptions (and their violations) look like.

Test your understanding!

Imagine you are building a model to predict the salary of a software developer. Your features are years_of_experience and lines_of_code_written_per_day.

1. Write down the formal equation for this linear regression model for a single developer $i$.
2. The "Zero Conditional Mean" assumption ($E({ϵ}_i|\mathbf{X})=0$) is critical. Can you describe a realistic factor that might be captured in the error term ${ϵ}_i$ and could violate this assumption? Explain why it would be a violation.

Show answer

1. Model Equation:

   $${\text{salary}}_i={\beta }_0+{\beta }_1({\text{years\_of\_experience}}_i)+{\beta }_2({\text{lines\_of\_code}}_i)+{ϵ}_i$$

2. Potential Violation: A very plausible omitted variable is the developer's innate problem-solving ability.

   • This "ability" is unobserved and is therefore part of the error term, ${ϵ}_i$.
   • It is likely correlated with salary (higher ability leads to higher salary).
   • It is also likely correlated with the included features. For example, a developer with higher ability might write more efficient (and perhaps fewer) lines of code to solve a problem. They might also be more likely to persist in the field, leading to more years of experience.
   • Because this unobserved factor ("ability") is part of ${ϵ}_i$ and is also correlated with the features in $\mathbf{X}$, the error term is no longer uncorrelated with the features. This means $E({ϵ}_i|\mathbf{X})\ne 0$, violating the Zero Conditional Mean assumption and leading to biased estimates of ${\beta }_1$ and ${\beta }_2$.

Conclusion

In this lesson, you have formally defined the linear regression model and, more importantly, delved into the rigorous statistical assumptions that underpin it. Understanding these assumptions is not just an academic exercise; it's what separates a mechanical model-fitter from a thoughtful data scientist who can critically evaluate a model's validity.

Key Takeaways:

• The linear regression model is expressed as $\mathbf{y}=\mathbf{X}\mathbf{\beta }+\mathbf{ϵ}$, where we aim to estimate the parameter vector $\mathbf{\beta }$.
• The model is linear in its parameters, but the features themselves can be non-linear.
• The Classical Linear Regression Model (CLRM) relies on a key set of assumptions:
  • Linearity in parameters
  • No perfect collinearity (full rank of $\mathbf{X}$)
  • Zero conditional mean / Strict exogeneity ($E(\mathbf{ϵ}|\mathbf{X})=0$): The most critical assumption for unbiasedness.
  • Spherical errors ($Var(\mathbf{ϵ}|\mathbf{X})={\sigma }^2{\mathbf{I}}_n$): Homoscedasticity and no autocorrelation, crucial for efficiency.
  • Normality of errors: Needed for valid statistical inference.

Preview of the Next Lesson:
Now that we have defined the model and its ideal conditions, the next logical question is: how do we find the "best" estimate for $\mathbf{\beta }$? In the next lesson, we will begin to answer this by deriving the cost function for linear regression from a least-squares perspective. This will set up the optimization problem that your calculus and optimization skills are perfectly prepared to solve.