Conditional Expectation and Information

Hello! Welcome to your next lesson.

Introduction

In our last lesson, we delved into the Strong Law of Large Numbers (SLLN). We saw that its precise statement, $P({\lim }_{n\to \infty }{\bar{X}}_n=\mu )=1$, is fundamentally measure-theoretic. The concept of a $\sigma$-algebra is necessary to even define the event of convergence, and the notion of "almost sure convergence" is the probabilistic counterpart to the measure-theoretic idea of "pointwise convergence almost everywhere."

Today, we will tackle another concept that you have frequently encountered in econometrics and statistics, but which also has deep measure-theoretic roots: conditional expectation. Your experience with regression models ($E[Y|X]$) and time series analysis provides a strong intuitive basis for this idea. Our goal is to formalize that intuition.

This lesson addresses the learning outcome: Define conditional expectation with respect to a σ-algebra, and interpret the σ-algebra as the 'information' available. We will first build the intuition for what a $\sigma$-algebra represents, then use that to construct the general, abstract definition of conditional expectation, and finally connect it back to the familiar forms you've worked with.

1. The σ-Algebra as Information

The cornerstone of the modern definition of conditional expectation is the idea that a $\sigma$-algebra represents information. A $\sigma$-algebra $\mathcal{G}$ on a sample space $\Omega$ is a collection of subsets (events). If we are "given the information in $\mathcal{G}$", it means that for any outcome $\omega \in \Omega$, we can determine whether $\omega$ is in any given set $A\in \mathcal{G}$, but we cannot distinguish between outcomes that are not separated by some set in $\mathcal{G}$.

Let's consider a simple example: rolling a standard six-sided die, so $\Omega =\{1,2,3,4,5,6\}$.

• No Information: The trivial $\sigma$-algebra is ${\mathcal{G}}_0=\{\varnothing ,\Omega \}$. If you are given this information, you know only that some outcome occurred, which is no information at all.
• Full Information: The power set, $\mathcal{F}=\mathcal{P}(\Omega )$, contains all possible subsets of $\Omega$. If you are given this information, you can identify the exact outcome. For any $\omega$, you can check if it belongs to the singleton set $\{\omega \}$, which is in $\mathcal{F}$.
• Partial Information: Consider the $\sigma$-algebra representing knowledge of the outcome's parity:
  ${\mathcal{G}}_{\text{parity}}=\{\varnothing ,\{1,3,5\},\{2,4,6\},\Omega \}$
  If you are given the information in ${\mathcal{G}}_{\text{parity}}$ and the outcome is $\omega =4$, you can determine that the event "the roll was even" ($\{2,4,6\}$) occurred. However, you cannot determine if the event "the roll was a 4" ($\{4\}$) occurred, because $\{4\}$ is not in ${\mathcal{G}}_{\text{parity}}$. You can't distinguish between a 2, 4, or 6.

This idea of a $\sigma$-algebra representing distinguishable events is the key to the entire theory.

Given your background, you can think of this in the context of time series. The information available at time $t$ includes the history of the process up to that point. This "information set" can be formally modeled by a $\sigma$-algebra, ${\mathcal{F}}_t$. As time progresses, we gain more information, so ${\mathcal{F}}_t\subseteq {\mathcal{F}}_{t+1}$. We will explore this sequence of $\sigma$-algebras, called a filtration, in the next lesson.

2. From Elementary to Abstract Conditional Expectation

With the interpretation of a $\sigma$-algebra as information, we can now build a bridge from the elementary definition of conditional expectation to the general one.

You know from basic probability that for an event $B$ with $P(B)>0$, the conditional expectation of a random variable $X$ given $B$ is:
$E[X|B]=\frac{E[X\cdot 1_B]}{P(B)}=\frac{1}{P(B)}{\int }_{B}X(\omega )dP(\omega )$
This is simply the average value of $X$ over the set $B$.

Now, let's extend this. Suppose we have a finite partition of $\Omega$ into disjoint sets $B_1,B_2,\dots ,B_n$, and let $\mathcal{G}$ be the $\sigma$-algebra generated by this partition. The information in $\mathcal{G}$ tells us which of the $B_i$ our outcome $\omega$ belongs to, but nothing more.

What would be our best guess, or expectation, of $X$ given this information? If we know $\omega \in B_i$, our best guess for $X(\omega )$ is the average of $X$ over that set, which is $E[X|B_i]$. Since this is our guess for any $\omega$ in $B_i$, we can define a new random variable, which we'll call $Z$, that takes on this value for all outcomes within each part of the partition:
$Z(\omega )={\sum }_{i=1}^{n}E[X|B_i]\cdot 1_{B_i}(\omega )$
Notice two crucial things about this random variable $Z$:

1. It is $\mathcal{G}$-measurable. Its value is constant on the atoms of $\mathcal{G}$ (the sets $B_i$), so if you have the information in $\mathcal{G}$, you know the value of $Z$.
2. It has a special averaging property. If we take any set $A\in \mathcal{G}$ (which must be a union of some of the $B_i$), the expectation of $Z$ over $A$ is the same as the expectation of $X$ over $A$. That is, ${\int }_{A}ZdP={\int }_{A}XdP$.

This second property turns out to be the defining characteristic of conditional expectation.

The main takeaway is that for a $\sigma$-algebra $\mathcal{G}$ generated by a finite partition, the conditional expectation $E[X|\mathcal{G}]$ can be identified as the unique $\mathcal{G}$-measurable random variable whose integral over any set in $\mathcal{G}$ matches that of $X$.

3. The Formal Definition

The insight from the partition example provides the blueprint for the general definition. We drop the requirement that $\mathcal{G}$ comes from a finite partition and take the integral property as our starting point.

Let $(\Omega ,\mathcal{F},P)$ be a probability space, $X$ an integrable random variable, and $\mathcal{G}$ a sub-$\sigma$-algebra of $\mathcal{F}$ ($\mathcal{G}\subseteq \mathcal{F}$).

We are looking for a random variable $Z$ that satisfies:

1. Measurability: $Z$ is $\mathcal{G}$-measurable (its value is determined by the information in $\mathcal{G}$).
2. Averaging Property: For every set $A\in \mathcal{G}$, ${\int }_{A}ZdP={\int }_{A}XdP$.

The existence and uniqueness (almost surely) of such a random variable $Z$ is guaranteed by a cornerstone of measure theory: the Radon-Nikodym Theorem.

Here's how it works, as sketched in the reading:

1. Define a new (signed) measure ${\mu }_X$ on the measurable space $(\Omega ,\mathcal{G})$ by ${\mu }_X(A)={\int }_{A}XdP$ for all $A\in \mathcal{G}$.
2. Observe that if $P(A)=0$, then ${\mu }_X(A)=0$. This property is called absolute continuity of ${\mu }_X$ with respect to $P$ (on $\mathcal{G}$).
3. The Radon-Nikodym theorem states that because ${\mu }_X$ is absolutely continuous with respect to $P$, there must exist a $\mathcal{G}$-measurable function—the Radon-Nikodym derivative $\frac{d{\mu }_X}{dP}$—such that for any $A\in \mathcal{G}$,
   ${\mu }_X(A)={\int }_A\frac{d{\mu }_X}{dP}dP$
4. Comparing this with our definitions, we see that ${\int }_{A}XdP={\int }_A\frac{d{\mu }_X}{dP}dP$. This derivative is exactly the random variable we were looking for.

We define the conditional expectation of $X$ given $\mathcal{G}$ as this Radon-Nikodym derivative.
$E[X|\mathcal{G}]:=\frac{d{\mu }_X}{dP}$

As the reading highlights, the two properties from Lemma A.1 are what you will use most often. A random variable $Z$ is a version of $E[X|\mathcal{G}]$ if and only if:

1. $Z$ is $\mathcal{G}$-measurable.
2. $E[Z\cdot 1_A]=E[X\cdot 1_A]$ for all $A\in \mathcal{G}$.

A crucial consequence, obtained by setting $A=\Omega$, is the Law of Total Expectation (or Tower Property):
$E[E[X|\mathcal{G}]]=E[X]$
This is the rigorous, general version of a rule you've likely used many times.

4. Connecting to Econometrics: Conditioning on a Random Variable

How does this abstract object, $E[X|\mathcal{G}]$, connect to the familiar $E[X|Y]$ from your econometrics courses?

The expression $E[X|Y]$ is simply shorthand for $E[X|\sigma (Y)]$, where $\sigma (Y)$ is the $\sigma$-algebra generated by the random variable $Y$.

• $\sigma (Y)$ is the smallest $\sigma$-algebra containing all sets of the form $\{\omega \in \Omega :Y(\omega )\in B\}$ for all Borel sets $B\subseteq \mathbb{R}$.
• Intuitively, $\sigma (Y)$ represents all the information obtained by observing the value of $Y$. Knowing the value of $Y$ is equivalent to having the information in $\sigma (Y)$.

From our definition, $E[X|\sigma (Y)]$ is a $\sigma (Y)$-measurable random variable. A fundamental result (the Doob-Dynkin lemma) states that any $\sigma (Y)$-measurable random variable can be written as a measurable function of $Y$. Therefore, there must exist some function $g:\mathbb{R}\to \mathbb{R}$ such that:
$E[X|\sigma (Y)]=g(Y)$
This random variable, $g(Y)$, is the formal object. The function $g(\cdot )$ is what we usually think of in elementary statistics. When we write $E[X|Y=y]$, we are referring to the value $g(y)$. The measure-theoretic definition gives us the entire random variable $g(Y)$ at once, without needing to worry about whether events like $\{Y=y\}$ have zero probability.

This generalization is incredibly powerful. It allows us to condition on entire collections of random variables, $\{Y_j{\}}_{j\in J}$, simply by using the $\sigma$-algebra they generate, $\sigma (\{Y_j{\}}_{j\in J})$. This is indispensable in stochastic calculus and the analysis of financial markets, where we might condition on the entire history of a price process.

Conclusion

Let's summarize the key takeaways from this lesson.

• Information as a σ-Algebra: A $\sigma$-algebra $\mathcal{G}$ provides a rigorous way to model a state of "information." It defines the collection of events that are distinguishable given that information.
• Conditional Expectation as a Random Variable: The conditional expectation of $X$ given $\mathcal{G}$, denoted $E[X|\mathcal{G}]$, is a $\mathcal{G}$-measurable random variable. It represents the best estimate of $X$ given only the information in $\mathcal{G}$.
• The Defining Property: Formally, $E[X|\mathcal{G}]$ is defined via the Radon-Nikodym theorem. In practice, it is characterized as the unique (a.s.) $\mathcal{G}$-measurable random variable $Z$ satisfying ${\int }_{A}ZdP={\int }_{A}XdP$ for all sets $A\in \mathcal{G}$.
• A Powerful Generalization: This single, abstract definition unifies and generalizes all the elementary forms of conditioning you have encountered, such as conditioning on an event ($E[X|B]$) or on another random variable ($E[X|Y]$).

Preview of the Next Lesson:
We have now defined what it means to have "information" ($\mathcal{G}$) and to take an "expectation given that information" ($E[\cdot |\mathcal{G}]$). The next logical step is to model how information evolves over time. We will introduce the concept of a filtration, which is an increasing sequence of $\sigma$-algebras. This will provide the formal framework needed to define and analyze stochastic processes, leading us directly to the measure-theoretic definition of a martingale.