Covariance and Correlation: Definitions and Properties

Recommended resources

The freely available resources this lesson will direct you to.

Hello! Welcome to our next lesson.

In our previous session, we established how to compute marginal and conditional distributions from a joint distribution. This gives us a complete, but often complex, picture of the relationship between random variables. Sometimes, however, we need a simpler, single-number summary of their relationship.

Today, we will delve into two of the most common metrics for this purpose: covariance and correlation. Your learning outcome is to define covariance and correlation, and prove their key properties. We will explore what these measures tell us about the relationship between variables, and just as importantly, what they don't tell us. This lesson will be rich with proofs, aligning with your goal of building a rigorous mathematical foundation.

1. Covariance: Measuring Joint Variability

Covariance measures how two random variables change together. Do they tend to be above their respective means at the same time? Or does one tend to be above its mean when the other is below?

1.1. Definition and Interpretation

Let $X$ and $Y$ be two random variables with means $μ_{X} = \E [X]$ and $μ_{Y} = \E [Y]$ , respectively.

The covariance between $X$ and $Y$ is defined as:
$\cov (X, Y) = \E [(X - μ_{X}) (Y - μ_{Y})]$

Let's break down the term inside the expectation, $(X - μ_{X}) (Y - μ_{Y})$ :

If $X$ and $Y$ are both above their means, this product is positive.
If $X$ and $Y$ are both below their means, this product is also positive.
If one is above its mean and the other is below, the product is negative.

The covariance is the average of this product over all possible outcomes.

$\cov (X, Y) > 0$ : Positive covariance. This suggests that as $X$ tends to be above its mean, $Y$ also tends to be above its mean. They are positively correlated.
$\cov (X, Y) < 0$ : Negative covariance. This suggests that as $X$ tends to be above its mean, $Y$ tends to be below its mean. They are negatively correlated.
$\cov (X, Y) = 0$ : Zero covariance. There is no linear tendency for them to vary together. They are uncorrelated.

1.2. Properties of Covariance and Their Proofs

The definition of covariance is intuitive, but not always the easiest to compute with. Let's derive some fundamental properties. As you requested, we will focus on the formal proofs for each.

First, let's watch a video that derives a few key properties using the linearity of expectation, a tool we've used many times.

L12.6 Covariance Properties

This video from MIT OpenCourseWare provides clear, step-by-step derivations for several core properties of covariance.

Please watch the entire video (00:00 - 05:47). It covers the following proofs: That the covariance of a variable with itself is its variance. The derivation of a more convenient computational formula for covariance. How covariance behaves under linear transformations (scaling and shifting). The linearity of the covariance operator.

The video you just watched established several key properties. Let's formally list them and ensure we understand the proofs. The following document provides a concise summary and proofs for these properties.

Reading 7b: Covariance and Correlation

This reading from an MIT course provides rigorous proofs for the key properties of covariance. It's an excellent reference to solidify your understanding.

Please read 'Section 2: Covariance' (page 1) and 'Section 2.4: Proofs of the properties of covariance' (page 4). Focus on the following six properties and their formal proofs: Behavior under linear transformation Additivity (bilinearity) Cov(X, X) = Var(X) Cov(X, Y) = E(XY) - E(X)E(Y) (the computational formula) Var(X + Y) = Var(X) + Var(Y) + 2Cov(X, Y) Independence implies zero covariance.

Let's recap the most important properties and their implications:

Computational Formula: $\cov (X, Y) = \E [X Y] - \E [X] \E [Y]$ . This is almost always how covariance is calculated in practice.
Variance of a Sum: $\var (X + Y) = \var (X) + \var (Y) + 2 \cov (X, Y)$ . This formula is crucial. It shows that variance is only additive for uncorrelated random variables (where $\cov (X, Y) = 0$ ). In machine learning, this appears when analyzing the variance of model parameters or errors that are sums of random components.
Independence implies Uncorrelated: If $X$ and $Y$ are independent, then $\E [X Y] = \E [X] \E [Y]$ , which directly leads to $\cov (X, Y) = 0$ . This is a fundamental result.

1.3. A Critical Warning: Uncorrelated $\neq =$ Independent

The converse of property #3 is not true. Zero covariance does not imply independence. Covariance only measures the linear relationship between variables. It can completely miss a strong non-linear relationship.

To see this in action, let's examine a classic counterexample.

Reading 7b: Covariance and Correlation

The same document provides a perfect example of two variables that are completely dependent yet have zero covariance.

Please read 'Example 2' on page 5. In this example, X is a simple discrete random variable and Y = X^2. Notice that they are clearly dependent, yet a direct calculation shows their covariance is zero.

This example is extremely important. Here, $Y$ is perfectly determined by $X$ . Yet, because the relationship is quadratic and symmetric around 0, their linear association is nonexistent, resulting in $\cov (X, Y) = 0$ . Always remember this limitation; it's a common pitfall.

2. Correlation: A Normalized Measure of Association

A major drawback of covariance is that its magnitude depends on the units of the variables. If you measure height in meters, the covariance between height and weight will be much smaller than if you measure height in millimeters, even though the underlying relationship is identical.

To fix this, we normalize the covariance.

2.1. Definition

The correlation coefficient (or simply correlation) of two random variables $X$ and $Y$ , denoted $ρ (X, Y)$ or $\cor (X, Y)$ , is defined as:
$ρ (X, Y) = \frac{\cov ( X , Y )}{σ _{X} σ _{Y}} = \frac{\cov ( X , Y )}{\var ( X ) \var ( Y )}$
where $σ_{X}$ and $σ_{Y}$ are the standard deviations of $X$ and $Y$ .

By dividing by the standard deviations, the units in the numerator (units of X $\times$ units of Y) are canceled out by the units in the denominator. This makes correlation a dimensionless quantity, allowing for a standardized comparison of linear association across different pairs of variables.

2.2. Properties of Correlation and Their Proofs

The most important property of correlation is its range.

Property 1: The correlation coefficient is bounded between -1 and 1.
$- 1 \leq ρ (X, Y) \leq 1$

Proof:
This proof is elegant and relies on the fact that variance can never be negative. Consider two new random variables, $U$ and $V$ , which are the standardized versions of $X$ and $Y$ :
$U = \frac{X - μ _{X}}{σ _{X}}, V = \frac{Y - μ _{Y}}{σ _{Y}}$
Note that $\E [U] = \E [V] = 0$ and $\var (U) = \var (V) = 1$ .

Let's compute the covariance of these standardized variables:
$\cov (U, V) = \E [U V] - \E [U] \E [V] = \E [U V] = \E [\frac{( X - μ _{X} )}{σ _{X}} \frac{( Y - μ _{Y} )}{σ _{Y}}]$
$\cov (U, V) = \frac{1}{σ _{X} σ _{Y}} \E [(X - μ_{X}) (Y - μ_{Y})] = \frac{\cov ( X , Y )}{σ _{X} σ _{Y}} = ρ (X, Y)$
So, the correlation of $X$ and $Y$ is simply the covariance of their standardized versions.

Now, let's look at the variance of their sum, $\var (U + V)$ . We know variance must be non-negative:
$0 \leq \var (U + V) = \var (U) + \var (V) + 2 \cov (U, V)$
Substituting what we know:
$0 \leq 1 + 1 + 2 ρ (X, Y) ⟹ 0 \leq 2 + 2 ρ ⟹ - 1 \leq ρ (X, Y)$

Similarly, let's look at the variance of their difference, $\var (U - V)$ :
$0 \leq \var (U - V) = \var (U) + \var (V) - 2 \cov (U, V)$
Substituting again:
$0 \leq 1 + 1 - 2 ρ (X, Y) ⟹ 0 \leq 2 - 2 ρ ⟹ ρ (X, Y) \leq 1$
Combining these two results gives the desired property: $- 1 \leq ρ (X, Y) \leq 1$ . This proof is also shown in the MIT reading LINK in section 3.2.

Property 2: Perfect Linear Relationship
$ρ (X, Y) = 1$ or $ρ (X, Y) = - 1$ if and only if $Y$ is a perfect linear function of $X$ (with probability 1). That is, $Y = a X + b$ for some constants $a \neq = 0$ and $b$ .

Proof Sketch:
This follows from the proof above. The only way for $\var (U - V)$ or $\var (U + V)$ to be exactly 0 is if the random variable itself is a constant (with probability 1).

If $ρ = 1$ , then $\var (U - V) = 0$ , which means $U - V = c$ for some constant $c$ . Substituting back the expressions for $U$ and $V$ and rearranging gives a linear relationship between $Y$ and $X$ with a positive slope.
If $ρ = - 1$ , then $\var (U + V) = 0$ , which means $U + V = c$ . This leads to a linear relationship with a negative slope.

This property solidifies the interpretation of correlation: a value of +1 or -1 indicates a perfect linear relationship, while a value of 0 indicates the absence of a linear relationship. Values in between indicate the strength of the linear trend.

For a deeper look at these properties and how correlation behaves under linear transformations (it is invariant to positive scaling and shifts), please consult the following resource.

4.5: Covariance and Correlation

This resource from LibreTexts provides further details on the properties of correlation, including a formal proof of its bounds derived from the concept of a 'best linear predictor' and its behavior under linear transformations.

Please read the subsections 'Properties of Correlation' and 'Additional properties of covariance and correlation'. Pay special attention to: How correlation is affected by linear transformations of the variables. The statement that \cor(X, Y) = \pm 1 if and only if there is a perfect linear relationship between X and Y.

Test your understanding!

Let $X$ be a random variable with $\var (X) > 0$ . Let $Y = - 5 X + 3$ . Without doing any complex calculations, what is $\cor (X, Y)$ ? Justify your answer based on the properties we have discussed.

Show answer

The correlation coefficient $\cor (X, Y)$ is -1.

Justification:
The relationship between $Y$ and $X$ is a perfect linear function: $Y = a X + b$ with $a = - 5$ and $b = 3$ . According to the properties of correlation, if two variables have a perfect linear relationship, their correlation must be either +1 or -1.

Because the slope $a = - 5$ is negative, the correlation must be -1.

Conclusion

In this lesson, we have formally defined covariance and correlation and rigorously proven their most important properties. These metrics are fundamental in statistics and machine learning, offering a first-pass quantitative look at the relationship between variables, for example, in exploratory data analysis or feature selection.

Key Takeaways:

Covariance, $\cov (X, Y) = \E [(X - μ_{X}) (Y - μ_{Y})]$ , measures the direction of the linear relationship between two variables. Its magnitude is dependent on the variables' units.
Correlation, $ρ (X, Y) = \frac{\cov ( X , Y )}{σ _{X} σ _{Y}}$ , is a normalized version of covariance that is dimensionless and bounded between -1 and 1. It measures both the direction and the strength of the linear relationship.
The crucial property $\var (X + Y) = \var (X) + \var (Y) + 2 \cov (X, Y)$ shows how covariance adjusts the variance of a sum.
A core principle to remember is that independence implies zero correlation, but zero correlation does NOT imply independence. Covariance and correlation only capture linear dependencies.

Preview of the Next Lesson:
We have now covered the essentials of describing and summarizing distributions of one or more random variables. Our next lesson will introduce two of the most powerful and fundamental theorems in all of probability theory: the Law of Large Numbers (LLN) and the Central Limit Theorem (CLT). These theorems describe the long-term behavior of sample averages and form the bedrock of most statistical inference and hypothesis testing.