Marginal and Conditional Distributions

Recommended resources

The freely available resources this lesson will direct you to.

Hello! Let's continue our journey into the mathematical foundations of AI.

In the last lesson, we formally defined joint, marginal, and conditional distributions, establishing the theoretical framework for describing relationships between multiple random variables. We saw that for any set of variables, the joint distribution contains all the probabilistic information, from which the marginal and conditional distributions can be derived.

Today, we transition from definition to application. Your learning outcome is to compute marginal and conditional distributions from a given joint distribution. This is a crucial skill that involves putting the formulas from our previous lesson into practice. We will work through concrete examples for both discrete and continuous random variables, requiring careful application of summation and integration techniques.

1. The Discrete Case: From Joint PMF to All Others

Let's begin with discrete random variables. As we saw last time, their behavior can often be described completely by a table representing their joint probability mass function (PMF). From this table, we can derive everything else.

The process is as follows:

Compute Marginal PMFs: To find the marginal PMF of one variable (e.g., $p_{X} (x)$ ), we sum the joint probabilities over all possible values of the other variable (e.g., $\sum_{y} p_{X, Y} (x, y)$ ). Geometrically, this corresponds to summing the values along the rows or columns of the joint probability table.
Compute Conditional PMFs: To find the conditional PMF (e.g., $p_{Y ∣ X} (y ∣ x)$ ), we use the fundamental definition of conditional probability:
$p_{Y ∣ X} (y ∣ x) = \frac{p _{X, Y} ( x , y )}{p _{X} ( x )}$
This involves dividing the joint probability by the marginal probability of the conditioning variable.

Let's apply this to a worked example.

Lecture 8: Joint Probability Distributions

The following document provides a clear, table-based example involving two discrete random variables. We will use this to practice computing marginal and conditional PMFs.

Please read 'Example 3' on page 14. This example defines a joint PMF table for two random variables, X and Y, representing the number of hoses in use at a service station. Familiarize yourself with the setup and the joint probability values in the table.

Now, let's perform the calculations based on that example.

Calculating Marginal Distributions

The marginal PMF of $X$ , $p_{X} (x)$ , is found by summing the probabilities in each row:

$p_{X} (0) = P (X = 0) = 0.10 + 0.04 + 0.02 = 0.16$
$p_{X} (1) = P (X = 1) = 0.08 + 0.20 + 0.06 = 0.34$
$p_{X} (2) = P (X = 2) = 0.06 + 0.14 + 0.30 = 0.50$

As a sanity check, these probabilities must sum to 1: $0.16 + 0.34 + 0.50 = 1.0$ .

Similarly, the marginal PMF of $Y$ , $p_{Y} (y)$ , is found by summing the probabilities in each column:

$p_{Y} (0) = P (Y = 0) = 0.10 + 0.08 + 0.06 = 0.24$
$p_{Y} (1) = P (Y = 1) = 0.04 + 0.20 + 0.14 = 0.38$
$p_{Y} (2) = P (Y = 2) = 0.02 + 0.06 + 0.30 = 0.38$

Again, checking the sum: $0.24 + 0.38 + 0.38 = 1.0$ .

Calculating Conditional Distributions

Now, let's find the conditional distribution of $Y$ given that $X = 2$ . This tells us the probability distribution for the number of hoses in use on the full-service island, given that we know exactly two hoses are in use on the self-service island.

We use the formula $p_{Y ∣ X} (y ∣2) = \frac{p _{X, Y} ( 2 , y )}{p _{X} ( 2 )}$ . We already calculated $p_{X} (2) = 0.50$ .

$p_{Y ∣ X} (0∣2) = \frac{p _{X, Y} ( 2 , 0 )}{p _{X} ( 2 )} = \frac{0.06}{0.50} = 0.12$
$p_{Y ∣ X} (1∣2) = \frac{p _{X, Y} ( 2 , 1 )}{p _{X} ( 2 )} = \frac{0.14}{0.50} = 0.28$
$p_{Y ∣ X} (2∣2) = \frac{p _{X, Y} ( 2 , 2 )}{p _{X} ( 2 )} = \frac{0.30}{0.50} = 0.60$

Notice that the resulting conditional probabilities sum to 1 ( $0.12 + 0.28 + 0.60 = 1.0$ ), as they must for any valid PMF. This new distribution, ${0.12, 0.28, 0.60}$ , fully describes the probabilistic behavior of $Y$ under the condition that $X = 2$ .

Test your understanding!

Using the same joint PMF table, calculate the conditional probability distribution of $X$ given that $Y = 1$ , i.e., find $p_{X ∣ Y} (x ∣1)$ for $x = 0, 1, 2$ .

Show answer

We need to compute $p_{X ∣ Y} (x ∣1) = \frac{p _{X, Y} ( x , 1 )}{p _{Y} ( 1 )}$ .
First, we recall the marginal probability $p_{Y} (1) = 0.38$ .

Now we compute the conditional probabilities for each value of $x$ :

$p_{X ∣ Y} (0∣1) = \frac{p _{X, Y} ( 0 , 1 )}{p _{Y} ( 1 )} = \frac{0.04}{0.38} \approx 0.105$
$p_{X ∣ Y} (1∣1) = \frac{p _{X, Y} ( 1 , 1 )}{p _{Y} ( 1 )} = \frac{0.20}{0.38} \approx 0.526$
$p_{X ∣ Y} (2∣1) = \frac{p _{X, Y} ( 2 , 1 )}{p _{Y} ( 1 )} = \frac{0.14}{0.38} \approx 0.368$

The sum of these probabilities is approximately 1, accounting for rounding. This is the PMF of $X$ given we know that $Y = 1$ .

2. The Continuous Case: Integration and Careful Limits

For continuous variables, the principles are identical, but summation is replaced by integration.

Compute Marginal PDFs: To find $f_{X} (x)$ , we integrate the joint PDF over all possible values of the other variable: $f_{X} (x) = \int_{- \infty}^{\infty} f_{X, Y} (x, y) d y$ .
Compute Conditional PDFs: The formula remains a ratio: $f_{Y ∣ X} (y ∣ x) = \frac{f _{X, Y} ( x , y )}{f _{X} ( x )}$ .

The primary challenge in the continuous case is correctly identifying the limits of integration. The region where the joint PDF is non-zero (the support of the distribution) determines these limits.

Let's begin by reviewing the general procedure for finding marginal and conditional PDFs from a joint PDF.

Lecture 19: Joint, Conditional, and Marginal Distributions | Statistics 110

This video from Harvard's Statistics 110 course clearly derives the formulas for marginalization and conditioning in the continuous case. It serves as a great warm-up for our next calculation.

Please watch from 06:15 to 11:37. This segment covers: Marginalization (06:15 - 08:36): How to obtain a marginal PDF by 'integrating out' the other variable. Conditioning (08:36 - 11:37): The definition of the conditional PDF as the ratio of the joint PDF to the marginal PDF.

Now, let's apply these formulas to a problem where the integration limits are not constant.

Lecture 13: Conditional Distributions and Joint Continuity

This document presents a problem where the joint PDF is uniform over a non-rectangular (triangular) region. This is a classic example that requires careful handling of integration limits.

Please read the 'Example' on page 13-5 and the calculations on page 13-6. The joint PDF is given as f_{X,Y}(x,y) = 1 over the triangular region shown in Figure 13.3. Focus on understanding how the marginal and conditional PDFs are derived step-by-step.

Let's break down those calculations. The joint PDF is $f_{X, Y} (x, y) = 1$ inside the triangle defined by the vertices (0,0), (1,0), and (0,2), and 0 elsewhere. The hypotenuse of this triangle is the line $y = - 2 x + 2$ , or $x = 1 - y /2$ .

Calculating Marginal PDFs

Marginal PDF of Y, $f_{Y} (y)$ :
To find $f_{Y} (y)$ , we integrate with respect to $x$ . For a fixed value of $y$ between 0 and 2, $x$ ranges from 0 to the line $x = 1 - y /2$ .
$f_{Y} (y) = \int_{- \infty}^{\infty} f_{X, Y} (x, y) d x = \int_{0}^{1 - y /2} 1 d x$
$f_{Y} (y) = [x]_{0}^{1 - y /2} = 1 - \frac{y}{2}, for 0 \leq y \leq 2$
And $f_{Y} (y) = 0$ otherwise.

Marginal PDF of X, $f_{X} (x)$ :
To find $f_{X} (x)$ , we integrate with respect to $y$ . For a fixed value of $x$ between 0 and 1, $y$ ranges from 0 to the line $y = 2 - 2 x$ .
$f_{X} (x) = \int_{- \infty}^{\infty} f_{X, Y} (x, y) d y = \int_{0}^{2 - 2 x} 1 d y$
$f_{X} (x) = [y]_{0}^{2 - 2 x} = 2 - 2 x, for 0 \leq x \leq 1$
And $f_{X} (x) = 0$ otherwise.

Calculating Conditional PDFs

Now we can find the conditional distributions by division.

Conditional PDF of X given Y, $f_{X ∣ Y} (x ∣ y)$ :
$f_{X ∣ Y} (x ∣ y) = \frac{f _{X, Y} ( x , y )}{f _{Y} ( y )} = \frac{1}{1 - y /2}, for 0 \leq x \leq 1 - y /2$
This result is fascinating. For a fixed value of $y$ , the conditional distribution of $X$ is uniform on the interval $[0, 1 - y /2]$ . The length of this interval depends on $y$ , but the density over it is constant with respect to $x$ .

Conditional PDF of Y given X, $f_{Y ∣ X} (y ∣ x)$ :
$f_{Y ∣ X} (y ∣ x) = \frac{f _{X, Y} ( x , y )}{f _{X} ( x )} = \frac{1}{2 - 2 x}, for 0 \leq y \leq 2 - 2 x$
Similarly, given a fixed $x$ , the conditional distribution of $Y$ is uniform on the interval $[0, 2 - 2 x]$ .

Test your understanding!

Given that we observe $X = 0.5$ , what is the probability that $Y > 0.5$ ? In other words, calculate $P (Y > 0.5∣ X = 0.5)$ .

Show answer

We first need the conditional PDF $f_{Y ∣ X} (y ∣0.5)$ . Using the formula we just derived:
$f_{Y ∣ X} (y ∣0.5) = \frac{1}{2 - 2 ( 0.5 )} = \frac{1}{1} = 1$
This conditional density is valid for $0 \leq y \leq 2 - 2 (0.5)$ , which is the interval $[0, 1]$ .

Now, we integrate this conditional PDF from 0.5 to the upper limit of its support, which is 1:
$P (Y > 0.5∣ X = 0.5) = \int_{0.5}^{1} f_{Y ∣ X} (y ∣0.5) d y = \int_{0.5}^{1} 1 d y = [y]_{0.5}^{1} = 1 - 0.5 = 0.5$
So, if the x-coordinate is 0.5, there is a 50% chance that the y-coordinate is greater than 0.5.

Conclusion

Today's lesson was purely computational, designed to solidify your understanding of the definitions from last time. You have practiced the two fundamental procedures for moving from a joint distribution to marginal and conditional ones.

Key Takeaways:

Marginal distributions are found by eliminating other variables from the joint distribution. This is done by summing over all values for discrete variables and integrating over the entire range for continuous variables.
Conditional distributions are found by taking the ratio of the joint distribution to the marginal distribution of the conditioning variable(s).
For continuous variables, the most critical step is correctly identifying the limits of integration, which are defined by the support of the joint distribution.

Preview of the Next Lesson:
While conditional and marginal distributions give us a complete picture, sometimes we need a simpler, single-number summary of the relationship between two variables. In the next lesson, we will introduce covariance and correlation. These metrics quantify the direction and strength of a linear relationship between two random variables, providing a first look into how one variable tends to change as another does.