Expected Value, Variance, and Covariance

Recommended resources

The freely available resources this lesson will direct you to.

Introduction

In our last lesson, we established the fundamental concept of a random variable, distinguishing between discrete variables described by Probability Mass Functions (PMFs) and continuous variables described by Probability Density Functions (PDFs). These functions provide a complete picture of a variable's probabilistic behavior. However, for many applications in machine learning and graphics, we need concise summaries of these distributions.

This lesson focuses on computing the three most important summary statistics: expectation, variance, and covariance. You will learn to calculate a distribution's "center of mass" (expectation), its "spread" (variance), and a measure of how two variables fluctuate together (covariance). These concepts are not just descriptive; they are the operational core of countless algorithms. For example, minimizing a loss function in deep learning is often equivalent to minimizing an expected value, and understanding the covariance structure of your data is key to techniques from Principal Component Analysis (PCA) to building generative models.

Expectation: The Center of Mass

The expectation (or expected value) of a random variable is its probability-weighted average. You can think of it as the "center of mass" of the distribution. If you were to repeat an experiment many times and average the outcomes, you would expect the average to converge to this value.

Formally, the expectation of a random variable $X$ , denoted $E [X]$ or $μ$ , is defined differently for discrete and continuous variables, mirroring the sum-integral divide we saw with PMFs and PDFs.

For a discrete random variable $X$ with PMF $p_{X} (x)$ : $E [X] = x \sum x \cdot p_{X} (x)$
For a continuous random variable $X$ with PDF $f_{X} (x)$ : $E [X] = \int_{- \infty}^{\infty} x \cdot f_{X} (x) d x$

A particularly powerful concept is the expectation of a function of a random variable, $g (X)$ . We can find its expectation without first finding the distribution of $g (X)$ . This principle is sometimes called the Law of the Unconscious Statistician (LOTUS).

Discrete: $E [g (X)] = \sum_{x} g (x) p_{X} (x)$
Continuous: $E [g (X)] = \int_{- \infty}^{\infty} g (x) f_{X} (x) d x$

The following reading from Stanford's CS229 course notes provides a concise, formal definition of expectation.

Review of Probability Theory - CS229

This section formally defines the expected value for both discrete and continuous random variables.

Please read the part of Section 2.4 titled Expectation. Pay attention to how the expectation is a "weighted average" and how the expectation of the variable itself, E[X], is a special case.

One of the most critical properties of expectation is its linearity. For any two random variables $X$ and $Y$ (even if they are dependent) and any constants $a, b \in R$ , we have:

E [a X + bY] = a E [X] + b E [Y]

This property is deceptively simple but incredibly powerful. It allows us to break down the expectation of a complex variable into the sum of expectations of simpler parts. This is a cornerstone of many proofs and derivations in machine learning.

Now for a question. We've seen that expectation distributes over sums. Does it also distribute over products? That is, is it always true that $E [X Y] = E [X] E [Y]$ ? If not, under what condition might this property hold?

Hint

Think back to our first lesson on probability. What was the special condition under which $P(A \cap B) = P(A)P(B)$? A similar concept applies here.

Answer

The property $E[XY] = E[X]E[Y]$ holds if and only if the random variables $X$ and $Y$ are **independent**. For dependent variables, this is generally not true. The difference between $E[XY]$ and $E[X]E[Y]$ will form the basis of our next major concept: covariance.

Variance: Measuring the Spread

While expectation tells us the center of a distribution, it doesn't tell us how "spread out" the values are. For instance, a random variable that is always exactly 50 has an expectation of 50. A variable that is 0 half the time and 100 the other half also has an expectation of 50. We need a way to quantify this spread.

This is the role of variance. The variance, denoted $V a r (X)$ or $σ^{2}$ , is the expected squared deviation of a random variable from its mean.

V a r (X) ≜ E [(X - E [X])^{2}]

A larger variance means the values are, on average, further from the mean. Since variance is a squared quantity, its units are the square of the original variable's units (e.g., meters-squared). To get back to the original units, we take the square root of the variance, which is called the standard deviation, $σ = V a r (X)$ .

Expanding the definition of variance using the linearity of expectation yields a very useful computational formula:

V a r (X) = E [(X - μ)^{2}] = E [X^{2} - 2 μ X + μ^{2}] = E [X^{2}] - E [2 μ X] + E [μ^{2}] (by linearity of expectation) = E [X^{2}] - 2 μ E [X] + μ^{2} (since μ is a constant) = E [X^{2}] - 2 μ (μ) + μ^{2} = E [X^{2}] - μ^{2} = E [X^{2}] - (E [X])^{2}

This formula, $V a r (X) = E [X^{2}] - (E [X])^{2}$ , is often easier to use in calculations than the original definition.

Expected Value and Variance

This resource provides a clear definition of variance and standard deviation, and derives the alternate computational formula.

Please read Section 6.2, "Variance of Discrete Random Variables." Focus on the definitions under Variance, and the derivation in Theorem 6.6. Although this section focuses on discrete variables, the definitions and properties are analogous for the continuous case.

A key property of variance relates to scaling and shifting. If $Y = a X + b$ , how is $V a r (Y)$ related to $V a r (X)$ ? You can derive this using the definition. A shift by $b$ moves the center, but does it change the spread? A scale by $a$ certainly changes the spread, but how?

Covariance: Measuring Joint Variability

We now have tools to describe the center and spread of a single variable. But in machine learning, we are almost always interested in the relationships between variables. Covariance is the tool that measures how two variables change together.

The covariance between two random variables $X$ and $Y$ is a generalization of variance:

C o v (X, Y) ≜ E [(X - E [X]) (Y - E [Y])]

If $C o v (X, Y) > 0$ , it indicates that when $X$ tends to be above its mean, $Y$ also tends to be above its mean (positive relationship).
If $C o v (X, Y) < 0$ , it indicates that when $X$ tends to be above its mean, $Y$ tends to be below its mean (negative relationship).
If $C o v (X, Y) = 0$ , the variables are said to be uncorrelated.

The intuition behind covariance is beautifully explained in the following video from Harvard's Stat 110 course.

Lecture 21: Covariance and Correlation | Statistics 110

This video introduces the definition of covariance and provides an intuitive explanation of what positive, negative, and zero covariance signify about the relationship between two random variables.

Watch the first part of the lecture from the beginning. This will ground your understanding of the formula in a clear, intuitive framework.

Similar to variance, covariance has a more convenient computational formula, which you can derive using linearity of expectation:

C o v (X, Y) = E [X Y] - E [X] E [Y]

Notice that if $X$ and $Y$ are independent, then $E [X Y] = E [X] E [Y]$ , which means their covariance is 0. Thus, independent variables are always uncorrelated. However, the reverse is not true! Two variables can be uncorrelated ( $C o v (X, Y) = 0$ ) but still be dependent. Covariance only measures linear relationships.

The most important application of covariance is in finding the variance of a sum of random variables. Unlike expectation, the variance of a sum is not simply the sum of the variances, unless the variables are uncorrelated. The general formula is:

V a r (X + Y) = V a r (X) + V a r (Y) + 2 C o v (X, Y)

This formula is a direct result of the properties of covariance. The following resource walks through the derivation of this and other essential properties.

Lecture 21: Covariance and Correlation | Statistics 110

This part of the lecture covers the crucial properties of covariance, including bilinearity, and uses them to derive the formula for the variance of a sum.

Watch the segment from Properties of Covariance. This section is dense but extremely important. Pay close attention to: The link between variance and covariance: Cov(X, X) = Var(X). Bilinearity: How covariance behaves with sums and constant multiples. The derivation of Var(X_1 + X_2).

Correlation and the Covariance Matrix

One issue with covariance is that its magnitude depends on the units of $X$ and $Y$ . A covariance of 100 might be small if the variables are measured in millions, but large if they are measured in small units. To create a standardized, unit-free measure of a linear relationship, we define the correlation coefficient, $ρ$ (rho).

ρ (X, Y) = \frac{C o v ( X , Y )}{σ _{X} σ _{Y}} = \frac{C o v ( X , Y )}{V a r ( X ) V a r ( Y )}

The correlation is always bounded between -1 and 1.

$ρ = 1$ : Perfect positive linear relationship.
$ρ = - 1$ : Perfect negative linear relationship.
$ρ = 0$ : No linear relationship (uncorrelated).

For your work in graphics and ML, you will often deal with data in high dimensions (e.g., pixel colors, vertex positions, feature vectors). We can generalize covariance to handle a vector of random variables $X = [X_{1}, X_{2}, \dots, X_{n}]^{T}$ . The relationship between all pairs of variables is captured in the covariance matrix, $Σ$ , an $n \times n$ matrix where:

Σ_{ij} = C o v (X_{i}, X_{j})

The diagonal entries $Σ_{ii}$ are simply the variances $V a r (X_{i})$ . The covariance matrix is symmetric ( $Σ = Σ^{T}$ ) and positive semidefinite. This matrix is fundamental to many multivariate methods, including the multivariate normal distribution and Principal Component Analysis (PCA).

Review of Probability Theory - CS229

This section extends the idea of covariance to multiple variables by defining the covariance matrix.

Please read the subsection titled Covariance matrix. Note how it's constructed and its final compact form, E[(X - E[X])(X - E[X])^T]. This matrix form will be essential when we discuss optimization and high-dimensional probability distributions.

Conclusion

In this lesson, we have equipped ourselves with the fundamental tools for summarizing probability distributions. These are not merely descriptive statistics; they are the gears of the machinery we will build.

Key Takeaways:

Expectation $E [X]$ is the probability-weighted average or center of mass of a distribution. It is a linear operator: $E [a X + bY] = a E [X] + b E [Y]$ .
Variance $V a r (X) = E [(X - E [X])^{2}]$ measures the spread of a distribution around its mean. A useful computational formula is $V a r (X) = E [X^{2}] - (E [X])^{2}$ .
Covariance $C o v (X, Y) = E [(X - E [X]) (Y - E [Y])]$ measures the linear relationship between two variables. Independence implies zero covariance, but not vice versa.
The variance of a sum depends on the covariance: $V a r (X + Y) = V a r (X) + V a r (Y) + 2 C o v (X, Y)$ .
Correlation is the normalized, unitless version of covariance, bounded between -1 and 1.
The Covariance Matrix generalizes covariance to vectors of random variables and is a cornerstone of multivariate statistics.

Preview of the Next Lesson:
We've defined the theoretical expected value and seen that it represents a long-run average. But how is this connection formalized? The next lesson introduces two of the most profound results in probability theory: the Law of Large Numbers (LLN) and the Central Limit Theorem (CLT). The LLN provides the guarantee that sample averages converge to the true expectation, justifying Monte Carlo methods used everywhere from financial modeling to photorealistic rendering. The CLT tells us how they converge, describing the shape of the error distribution, which is crucial for statistical inference and understanding uncertainty in our models.