Bias-Variance Tradeoff Decomposition

Setting the Stage for the Decomposition

Before we dive into the mathematics, let's establish the scenario. We are in a regression setting where we want to predict a continuous target variable, $y$, based on a set of features, $x$. We assume there's a true, but unknown, underlying function, $f(x)$, that governs this relationship. However, the data we observe is not perfect; it contains some inherent randomness, or noise. We can model this relationship as:

$y=f(x)+ϵ$

Here, $ϵ$ is a random noise term with a mean of zero ($\mathbb{E}[ϵ]=0$) and a variance of ${\sigma }^2$ ($\text{Var}(ϵ)={\sigma }^2$). This noise represents factors that influence $y$ but are not captured by our features $x$. The term ${\sigma }^2$ is often called the irreducible error because no model, no matter how powerful, can eliminate this fundamental uncertainty in the data-generating process.

Our task is to use a training dataset, $D=\{(x_1,y_1),\dots ,(x_n,y_n)\}$, to learn a model, which we'll call ${\hat{f}}_D(x)$, that provides the best possible estimate of $f(x)$.

A crucial point to grasp is that our trained model, ${\hat{f}}_D(x)$, is itself a random variable. Why? Because it was trained on the dataset $D$, which is a random sample from the true data distribution. If we were to draw a different training dataset, $D^{\prime }$, we would obtain a slightly different model, ${\hat{f}}_{D^{\prime }}(x)$. The bias-variance decomposition analyzes the error of our learning algorithm by considering its average performance over all possible training datasets that could have been drawn.

Our goal is to analyze the expected prediction error of our model at a new, unseen test point $x$. We'll use the Mean Squared Error (MSE) for this:

$\text{Error}(x)=\mathbb{E}\left[(y-{\hat{f}}_D(x){)}^2\right]$

The expectation $\mathbb{E}[\cdot ]$ is taken over both the randomness in the training data $D$ (which makes ${\hat{f}}_D$ random) and the randomness in the new observation $y$ (due to the noise term $ϵ$). It is this error term that we will now decompose.

The Mathematical Derivation

The derivation of the bias-variance decomposition is a classic and elegant piece of statistical theory. It relies on a clever algebraic trick of adding and subtracting the same terms.

The clearest way to see the proof is often in a live lecture format. Professor Kilian Weinberger's lecture provides an excellent, step-by-step walkthrough of the derivation from first principles.

Let's summarize the derivation you just saw. We start with the expected squared error and strategically add and subtract two key quantities:

1. ${\mathbb{E}}_D[{\hat{f}}_D(x)]$: The average prediction of our model at point $x$ over all possible training sets. Let's call this $\bar{f}(x)$.
2. $f(x)$: The true, underlying function.

The expression $\mathbb{E}\left[(y-{\hat{f}}_D(x){)}^2\right]$ can be expanded by substituting $y=f(x)+ϵ$ and adding and subtracting $\bar{f}(x)$:

When you expand this squared term, the cross-product terms conveniently go to zero under expectation, because:

• $\mathbb{E}[ϵ]=0$
• ${\hat{f}}_D(x)$ and $ϵ$ are independent.
• By definition, ${\mathbb{E}}_D[\bar{f}(x)-{\hat{f}}_D(x)]={\mathbb{E}}_D[\bar{f}(x)]-{\mathbb{E}}_D[{\hat{f}}_D(x)]=\bar{f}(x)-\bar{f}(x)=0$.

This leaves us with three distinct terms:

For a more leisurely, text-based walkthrough of this same proof, the following article is an excellent resource.

Interpreting the Three Components of Error

The decomposition reveals that the error of any machine learning model stems from three sources.

1. Bias (squared): ${\left({\mathbb{E}}_D[{\hat{f}}_D(x)]-f(x)\right)}^2$

   • What it is: Bias measures how far off the average prediction of our model is from the true value we are trying to predict. It reflects the error introduced by the model's underlying assumptions. A linear model, for instance, has a "bias" towards linear relationships and will produce systematic errors if the true function is quadratic.
   • Implication: High bias leads to underfitting. The model is too simple to capture the underlying structure of the data, and it consistently misses the target, even with more data.

2. Variance: ${\mathbb{E}}_D\left[({\hat{f}}_D(x)-{\mathbb{E}}_D[{\hat{f}}_D(x)]{)}^2\right]$

   • What it is: Variance measures how much the model's predictions for a given point $x$ fluctuate for different training datasets. It quantifies the model's sensitivity to the specific training data it received.
   • Implication: High variance leads to overfitting. The model is too complex and learns the noise and quirks of the specific training set. When presented with a new dataset, its performance is unstable and likely poor.

3. Irreducible Error (Noise): ${\sigma }^2$

   • What it is: This error is a property of the data itself, not the model. It's the inherent randomness that cannot be predicted.
   • Implication: It sets a hard lower limit on the performance of any predictive model.

This tradeoff is often visualized as a U-shaped curve, where total error is plotted against model complexity.

A classic analogy, which you saw in Professor Weinberger's lecture, is that of a dart thrower.

• Low Bias, Low Variance: A skilled dart thrower who consistently hits the bullseye. (The ideal model)
• High Bias, Low Variance: A consistent but inaccurate thrower who always hits the same spot on the board, but far from the bullseye. (Underfitting)
• Low Bias, High Variance: An unsteady thrower whose darts land all around the bullseye, but the average position is correct. (Overfitting)
• High Bias, High Variance: An inaccurate and unsteady thrower whose darts are scattered all over, far from the bullseye. (The worst case)

A Deeper Dive: Model Bias vs. Estimation Bias

For a more advanced perspective, especially relevant to your work in data science, we can even decompose the bias term further. This is particularly insightful when we use a simple model class (like linear regression) to approximate a potentially complex reality.

As you just read, the bias of a linear model can be broken down into:

• Model Bias: The error that arises because your chosen model class (e.g., linear models) might be fundamentally incapable of representing the true function $f(x)$. This is the error of the best possible linear model. If the true world is non-linear, even the best linear model will have some bias.
• Estimation Bias: The difference between the average model we learn from our finite data (${\mathbb{E}}_D[{\hat{f}}_D(x)]$) and the best possible model within our chosen class. For Ordinary Least Squares (OLS), this term is zero if the assumptions hold. However, techniques like Ridge Regression deliberately introduce a small amount of estimation bias in order to drastically reduce variance, a concept we will explore in detail later in this course.

Conclusion

In this lesson, we established the mathematical foundation of one of machine learning's most important concepts.

Key Takeaways:

• A model's expected error on new data can be decomposed into three components: $\text{Error}={\text{Bias}}^2+\text{Variance}+\text{Irreducible Error}$.
• Bias represents systematic error from a model's assumptions, leading to underfitting.
• Variance represents the model's sensitivity to the training data, leading to overfitting.
• Irreducible Error is the noise floor set by the data itself.
• The goal of model selection is to find a sweet spot of complexity that balances the tradeoff between bias and variance to minimize the total error.

This framework provides a powerful lens through which to analyze model performance. As you encounter different algorithms and tuning techniques throughout this course, always ask yourself: "How is this affecting the model's bias and variance?"

Having built this theoretical framework, our next lesson will shift to the practical side of model evaluation. We will explore key metrics for classification tasks, including precision, recall, F1-score, and the ROC-AUC, paying special attention to the challenges posed by imbalanced datasets.