Covariance and Correlation of Linear Combinations

Hello! Welcome to your next lesson in the "Probability Models for Data" module.

Introduction

In our previous lesson, we explored Markov's and Chebyshev's inequalities. These powerful tools demonstrated how the mean and, more importantly, the variance of a random variable allow us to place bounds on its behavior, even without knowing its exact distribution. We saw that variance is a fundamental measure of a variable's spread around its mean.

Today, we will delve deeper into the relationships between multiple random variables. Often in data science, we aren't just interested in a single variable, but in how several variables interact. For instance, how is an EV's battery range related to the ambient temperature? How does a change in one stock price relate to another in a portfolio?

To answer these questions, we need to introduce covariance and correlation. This lesson will equip you with the tools to analyze the variance of combinations of random variables. You will learn to:

Define covariance and correlation, both algebraically and geometrically.
Derive the variance for a sum of random variables, accounting for their relationship.
Generalize this to derive the variance and covariance for any linear combination of random variables.

This topic is foundational for many advanced concepts, including portfolio optimization, error propagation analysis in engineering systems, and dimensionality reduction techniques like Principal Component Analysis (PCA), which we will cover later in the course.

1. Covariance and Correlation: Quantifying Relationships

While variance measures how a single variable fluctuates, covariance measures how two variables fluctuate together.

Definition of Covariance

The covariance between two random variables $X$ and $Y$ is defined as the expected value of the product of their deviations from their respective means:
$C o v (X, Y) = E [(X - E [X]) (Y - E [Y])]$
Intuition:

If $X$ tends to be above its mean when $Y$ is above its mean, the term inside the expectation will likely be positive.
If $X$ tends to be above its mean when $Y$ is below its mean (or vice versa), the term will likely be negative.
The expected value, $C o v (X, Y)$ , is the average of this product over all possible outcomes. A positive covariance implies $X$ and $Y$ tend to move in the same direction, while a negative covariance implies they move in opposite directions.

A computationally useful form of the covariance is:
$C o v (X, Y) = E [X Y] - E [X] E [Y]$
You can derive this simply by expanding the first definition. Notice that if you set $Y = X$ , you get $C o v (X, X) = E [X^{2}] - (E [X])^{2}$ , which is precisely the definition of $V a r (X)$ . Variance is the covariance of a random variable with itself.

The Correlation Coefficient

A drawback of covariance is that its magnitude is dependent on the units of $X$ and $Y$ . To get a standardized, unit-free measure of the relationship, we use the correlation coefficient, denoted by $ρ$ (rho).

$ρ (X, Y) = \frac{C o v ( X , Y )}{σ _{X} σ _{Y}} = \frac{C o v ( X , Y )}{V a r ( X ) V a r ( Y )}$

This normalization constrains $ρ$ to the range $[- 1, 1]$ , where:

$ρ = + 1$ : Perfect positive linear relationship.
$ρ = - 1$ : Perfect negative linear relationship.
$ρ = 0$ : No linear relationship. This is a crucial point: zero correlation does not necessarily mean the variables are independent; it only means they are not linearly related.

A Geometric Interpretation

Given your background, you'll find the geometric interpretation of these concepts particularly elegant. Imagine that centered random variables (where the mean has been subtracted, e.g., $X^{'} = X - E [X]$ ) are vectors in a high-dimensional space defined by the outcomes of the experiment.

The covariance, $C o v (X, Y) = E [X^{'} Y^{'}]$ , is their dot product.
The variance, $V a r (X) = E [X^{'2}]$ , is the squared magnitude (length) of the vector $X^{'}$ .
The correlation, $ρ (X, Y)$ , is the cosine of the angle between the vectors $X^{'}$ and $Y^{'}$ .

This means that two uncorrelated random variables are geometrically orthogonal.

Lecture 12: Correlation

This PDF from a Harvard course on Linear Algebra with Probability provides a concise and clear explanation of this geometric viewpoint.

Read the first page of this handout. Focus on the section that begins 'Here is a key connection between linear algebra and probability theory'. This will solidify the analogy between covariance/correlation and dot products/angles.

The shape of a 2D data distribution is directly linked to its covariance matrix.

Geometric Interpretation of Covariance Matrix — This image illustrates how the covariance matrix affects a 2D distribution. A diagonal covariance matrix (zero covariance) results in axes-aligned contours (top row). Non-zero covariance (bottom row) indicates correlation and results in tilted elliptical contours. The shape and orientation of the ellipse are determined by the eigenvectors and eigenvalues of the covariance matrix, a concept we'll explore with PCA.

2. Variance of Linear Combinations

Now we'll address the main goal of the lesson. What is the variance of a sum, like $X + Y$ ? A common mistake is to assume $V a r (X + Y) = V a r (X) + V a r (Y)$ , but this is only true in a special case. Let's derive the general formula.

The variance of a sum of two random variables is:
$V a r (X + Y) = V a r (X) + V a r (Y) + 2 C o v (X, Y)$

The following video from MIT OpenCourseWare provides a clear, step-by-step derivation.

L12.7 The Variance of the Sum of Random Variables

Professor John Tsitsiklis derives the formula for the variance of a sum of two random variables, starting from the basic definition of variance.

Watch the segment from the beginning until 02:54. Follow how the expansion of the squared term naturally gives rise to the variance and covariance terms.

This formula shows that the variance of a sum depends critically on how the variables co-vary.

Positive Covariance: If $C o v (X, Y) > 0$ , the variables tend to move together. When $X$ is large, $Y$ is likely large, making the sum $X + Y$ very large. This increases the overall variance.
Negative Covariance: If $C o v (X, Y) < 0$ , the variables tend to move in opposite directions. A large $X$ is paired with a small $Y$ , so the sum $X + Y$ tends to stay moderate. This decreases the overall variance.
Zero Covariance: Only when $X$ and $Y$ are uncorrelated does the variance of the sum equal the sum of the variances. This is the "Pythagorean Theorem" for random variables that our geometric interpretation suggests.

For the variance of a difference, the sign flips:
$V a r (X - Y) = V a r (X) + V a r (Y) - 2 C o v (X, Y)$

3. The General Case: Scaling and Multiple Variables

Let's generalize this to any linear combination, such as $a X + bY + c$ . To do this, we first need to understand how covariance behaves under linear transformations. This property is known as bilinearity.

Properties of Covariance

Let $X, Y, U, V$ be random variables and $a, b, c, d$ be constants.

Symmetry: $C o v (X, Y) = C o v (Y, X)$
Self-covariance: $C o v (X, X) = V a r (X)$
Effect of Scaling and Shifting: $C o v (a X + b, c Y + d) = a c \cdot C o v (X, Y)$ . Note that additive constants $b$ and $d$ disappear, as they don't affect variability.

Variance of a General Linear Combination

Now we can derive the formula for $V a r (a X + bY)$ .
This video provides an excellent, self-contained proof of the general formula.

ECO375F - 2.5 - Important Variance and Covariance Rules

This video walks through the complete proof for the variance of a general linear combination, Var(aX + bY + c), using the basic definitions.

Watch the 'Proof' section from 05:02 to 11:16. The presenter systematically breaks down the problem, applying the rules of expectation and the definitions of variance and covariance. This is the core derivation for this lesson.

The final result is:
$V a r (a X + bY) = a^{2} V a r (X) + b^{2} V a r (Y) + 2 ab \cdot C o v (X, Y)$

Extension to More Than Two Variables

This logic extends to the sum of $n$ random variables. The variance of the sum is the sum of all their individual variances plus the sum of all their pairwise covariances (each multiplied by 2).
$V a r (\sum_{i = 1}^{n} X_{i}) = \sum_{i = 1}^{n} V a r (X_{i}) + \sum_{i \neq = j} C o v (X_{i}, X_{j})$
This can be written more cleanly as:
$V a r (\sum_{i = 1}^{n} X_{i}) = \sum_{i = 1}^{n} \sum_{j = 1}^{n} C o v (X_{i}, X_{j})$
since $C o v (X_{i}, X_{i}) = V a r (X_{i})$ .

For the most general case of a weighted sum, $L = \sum_{i = 1}^{n} a_{i} X_{i}$ , the formula becomes:
$V a r (L) = \sum_{i = 1}^{n} a_{i}^{2} V a r (X_{i}) + \sum_{i \neq = j} a_{i} a_{j} C o v (X_{i}, X_{j}) = \sum_{i = 1}^{n} \sum_{j = 1}^{n} a_{i} a_{j} C o v (X_{i}, X_{j})$

In matrix notation, this is expressed beautifully. If $a$ is a column vector of weights $[a_{1}, ..., a_{n}]^{T}$ , $X$ is a column vector of the random variables, and $Σ$ is the $n \times n$ covariance matrix where $Σ_{ij} = C o v (X_{i}, X_{j})$ , then:
$V a r (a^{T} X) = a^{T} Σa$
This compact form is extremely powerful and widely used in finance, engineering, and machine learning.

4. Practice Problem

Let's apply these concepts to a concrete example.

Expected values of linear combinations of random variables

This text provides an excellent applied problem involving a basketball player's points. It requires you to calculate the variance and standard deviation of linear combinations of multiple correlated random variables.

Navigate to section 5.5.4, 'Bilinearity of covariance'. Find Example 5.36 and try to solve all three parts of the problem on your own. Use the provided standard deviations and correlations to compute the covariances first. Then apply the general variance formula for a sum. After you have your answers, check them against the provided solution.

Conclusion

This lesson has been a deep dive into the mathematics of joint variability. We've moved beyond single variables to understanding how they interact and how that interaction affects the uncertainty of their combinations.

Key Takeaways:

Covariance measures the directional relationship between two variables, while correlation provides a standardized measure of the linear relationship, bounded between -1 and +1.
Geometrically, for centered random variables, covariance is the dot product, variance is squared magnitude, and correlation is the cosine of the angle between them.
The variance of a sum of random variables includes a covariance term: $V a r (X + Y) = V a r (X) + V a r (Y) + 2 C o v (X, Y)$ . Variances only add when the variables are uncorrelated.
For a general linear combination, $V a r (a X + bY) = a^{2} V a r (X) + b^{2} V a r (Y) + 2 ab C o v (X, Y)$ .
In matrix form, the variance of a linear combination $a^{T} X$ is given by $a^{T} Σa$ , where $Σ$ is the covariance matrix.

Preview of the Next Lesson:
We've established the rules for the mean and variance of sums of random variables. Next, we will investigate the behavior of these sums when the number of variables, $n$ , becomes very large. This will lead us to two cornerstone results of probability and statistics: the Law of Large Numbers and the Central Limit Theorem. We will formally derive the Weak Law of Large Numbers using Chebyshev's inequality, tying together several concepts from our recent lessons.

Recommended resources