Conditional Probability and Independence

The Idea of a Reduced Sample Space

Imagine you roll a standard six-sided die. The probability of rolling a 2 is $1/6$. But what if I tell you that I've already rolled the die and I can confirm the result is an even number? Does the probability of it being a 2 change?

Of course it does. The original sample space was $\{1,2,3,4,5,6\}$. With the new information, we can discard the odd numbers. Our "universe" of possible outcomes has shrunk; the new, reduced sample space is now just $\{2,4,6\}$. Within this new reality, the probability of the outcome being a 2 is now $1/3$.

This is the core intuition of conditional probability: new information restricts the sample space, which can alter the probabilities of events within it. The following video provides an excellent introduction to this idea.

This idea of restricting the sample space can be visualized effectively. If we think of our sample space as a large area, and events as regions within it, conditioning on an event B means we are now only concerned with the area inside B.

The Formal Definition of Conditional Probability

Our intuitive understanding—that we're looking at the fraction of the "new" sample space B that also corresponds to A—leads directly to the formal definition. The "part of B that is also A" is the intersection, $A\cap B$. The size of our new sample space is simply the probability of B, $P(B)$.

Therefore, the conditional probability of A given B, written as $P(A|B)$, is defined as:

This formula holds provided that $P(B)\ne 0$, as we cannot condition on an event that has no chance of happening.

The next resource provides a concise, formal definition and solidifies the concept using Venn diagrams and a clear mathematical derivation.

To see this formula in action, let's work through a classic example. The MIT OpenCourseWare notes below provide a clean, step-by-step application of the formula.

The Multiplication Rule

A simple rearrangement of the conditional probability formula gives us another crucial tool: the multiplication rule. If we multiply both sides of the definition by $P(B)$, we get:

This rule is our primary method for calculating the probability of the intersection of two events, especially in sequential processes. It essentially states that the probability of A and B happening is the probability of B happening, multiplied by the probability of A happening given that B already happened.

The MIT notes you just reviewed introduce this rule right after the definition of conditional probability. It's a direct and important consequence.

Independence: When Information Doesn't Matter

We began by seeing how information can change probability. But what if it doesn't?

Let's go back to the deck of cards.

• The unconditional probability of drawing a Jack is $P(\text{Jack})=4/52=1/13$.
• Now, I give you some information: the card drawn is red. What is the new probability?
  The reduced sample space is the 26 red cards. Among them, there are 2 Jacks (the Jack of Hearts and the Jack of Diamonds).
• So, the conditional probability is $P(\text{Jack}|\text{Red})=2/26=1/13$.

The probability didn't change! $P(\text{Jack}|\text{Red})=P(\text{Jack})$. The information that the card was red was irrelevant to the probability of it being a Jack. When this happens, we say the two events are independent.

Formally, two events A and B are independent if the occurrence of one does not affect the probability of the other. This gives us two equivalent ways to define independence:

1. Using Conditional Probability: A and B are independent if $P(A|B)=P(A)$ (and equivalently, $P(B|A)=P(B)$).
2. Using the Multiplication Rule: If we substitute $P(A|B)=P(A)$ into the multiplication rule, we get a simple and powerful test for independence: $$P(A\cap B)=P(A)P(B)$$ This is the most common technical definition of independence.

Let's explore this concept with a couple of clear examples.

Testing for Independence with Data

In data science, you often won't know the true theoretical probabilities. Instead, you'll have experimental data, typically organized in a table (what you might call a contingency table or a confusion matrix in other contexts). We can use this data to calculate experimental probabilities and test for independence.

The Khan Academy video below provides an excellent walkthrough of this exact process, which is a core skill in exploratory data analysis.

Visualizing with Probability Trees

When dealing with a sequence of events, a probability tree is an invaluable tool for organizing your thoughts and calculations.

• The first level of branches represents the outcomes of the first event, with probabilities on the branches.
• The second level of branches represents the outcomes of the second event, but the probabilities here are conditional probabilities.
• To find the probability of a complete path (e.g., event A followed by event B), you multiply the probabilities along the branches. This is a direct application of the multiplication rule: $P(A\cap B)=P(A)\times P(B|A)$.

This structure makes calculating the probabilities of complex sequences much more manageable.

Conclusion

In this lesson, we've explored the powerful concepts of conditional probability and independence. These ideas are the bedrock for more advanced topics like Bayes' theorem and are used constantly in machine learning models, from simple classifiers to complex probabilistic systems.

Key Takeaways:

• Conditional probability, $P(A|B)$, is the probability of event A occurring, given that event B has already occurred. It involves reasoning about a reduced sample space.
• The formal definition is $P(A|B)=\frac{P(A\cap B)}{P(B)}$.
• The Multiplication Rule, $P(A\cap B)=P(A|B)P(B)$, is used to find the probability of the intersection of two events.
• Two events A and B are independent if knowing one has occurred does not change the probability of the other.
• The two key tests for independence are $P(A|B)=P(A)$ and the more general $P(A\cap B)=P(A)P(B)$.
• Probability trees are an excellent way to visualize and calculate probabilities for sequential events.

In our next lesson, we will put all of our new probability tools—the addition rule, the multiplication rule, and conditional probability—to work by solving more complex applied problems that require combining these rules in a structured way.