Probability Calculations: Axioms, Conditional Probability, and Bayes' Theorem

1. The Language and Rules of Probability

Before we can calculate probabilities, we need a common language.

• Sample Space ($\Omega$): The set of all possible outcomes of an experiment. For a single coin flip, $\Omega =\{\text{Heads, Tails}\}$.
• Event (A): A subset of the sample space; a specific outcome or set of outcomes we are interested in. For a coin flip, an event might be "the coin lands on Heads."
• Probability P(A): A number between 0 and 1 assigned to an event A, representing its likelihood.

The most basic way to calculate probability is:
$P(\text{Event})=\frac{\text{Number of favorable outcomes}}{\text{Total number of outcomes in the sample space}}$

This is where our work from the last lesson comes in! You use combinations and permutations to find the numerator and denominator.

The Axioms of Probability

In the 1930s, the mathematician Andrey Kolmogorov established three simple but powerful rules, or axioms, upon which all of modern probability theory is built. Any system for assigning probabilities must obey these three rules.

To understand these axioms, let's watch a short, clear explanation from MIT OpenCourseWare.

To summarize and review, the three axioms are:

1. Non-negativity: The probability of any event is non-negative.
   $P(A)\ge 0$
2. Normalization: The probability of the entire sample space is 1. This means that one of the possible outcomes must occur.
   $P(\Omega )=1$
3. Additivity: If two events, A and B, are mutually exclusive (meaning they cannot happen at the same time, i.e., $A\cap B=\varnothing$), then the probability of A or B happening is the sum of their individual probabilities.
   $P(A\cup B)=P(A)+P(B)\text{if A and B are mutually exclusive}$

From these three simple axioms, we can derive many other useful properties of probability. The following reading provides a great summary of these axioms and derived rules, including the important General Addition Rule for events that are not mutually exclusive.

The General Addition Rule is a key takeaway from that reading:
$P(A\cup B)=P(A)+P(B)-P(A\cap B)$
We subtract the probability of the intersection, $P(A\cap B)$, to avoid double-counting the outcomes that belong to both events.

2. Conditional Probability: Probability with New Information

Often, we want to know how the likelihood of an event changes when we receive new information. For example:

• What is the probability a student will pass the GATE exam?
• What is the probability a student will pass, given that they studied 3 hours every day?

This is the domain of conditional probability. The notation is $P(A|B)$, which we read as "the probability of A given B." It measures the probability of event A happening, knowing that event B has already happened.

The formula for conditional probability is:
$P(A|B)=\frac{P(A\cap B)}{P(B)}$

Intuitively, since we know B has occurred, our world of possible outcomes shrinks from the entire sample space $\Omega$ down to just the outcomes in B. The "favorable" outcomes are those that are also in A, which is the intersection $A\cap B$.

Let's watch a video that makes this concept very clear.

A simple rearrangement of the conditional probability formula gives us the Multiplication Rule, which is essential for our next topic.
$P(A\cap B)=P(A|B)\cdot P(B)$

3. Bayes' Theorem: Updating Your Beliefs

Bayes' Theorem is one of the most important concepts in probability and a cornerstone of machine learning (e.g., in spam filters and medical diagnosis). It gives us a way to formally update our beliefs about a hypothesis in light of new evidence. In essence, it lets us "flip" a conditional probability. We often know $P(B|A)$ but want to find $P(A|B)$.

The Derivation

We start with the multiplication rule written in two ways:

1. $P(A\cap B)=P(A|B)\cdot P(B)$
2. $P(B\cap A)=P(B|A)\cdot P(A)$

Since $P(A\cap B)$ is the same as $P(B\cap A)$, we can set the right sides equal:
$P(A|B)\cdot P(B)=P(B|A)\cdot P(A)$

Dividing by $P(B)$, we get Bayes' Theorem:
$P(A|B)=\frac{P(B|A)\cdot P(A)}{P(B)}$

The Terminology

• $P(A|B)$ is the Posterior: The updated probability of A after considering the evidence B.
• $P(A)$ is the Prior: Our initial belief in the probability of A before seeing the evidence.
• $P(B|A)$ is the Likelihood: The probability of observing evidence B if our hypothesis A is true.
• $P(B)$ is the Evidence: The total probability of observing the evidence B.

A common challenge is calculating the evidence, $P(B)$. We often do this using the Law of Total Probability:
$P(B)=P(B|A)\cdot P(A)+P(B|A^c)\cdot P(A^c)$
This breaks down the probability of B into the cases where A happens and where A doesn't happen ($A^c$).

The following reading provides a detailed walkthrough of Bayes' Theorem and two excellent examples that are classic GATE-style problems.

The medical diagnosis problem is a classic application of Bayes' Theorem. Let's frame it as an exercise.

Example Problem (Disease Testing)
A certain disease affects 1% of the population. A test for the disease is 95% accurate if you have the disease (it gives a positive result 95% of the time). The test is 90% accurate if you do not have the disease (it gives a negative result 90% of the time).

A person takes the test and gets a positive result. What is the actual probability that they have the disease?

Let's define our events:

• D = Person has the disease
• T = Test result is positive

We want to find $P(D|T)$.

From the problem statement, we know:

• $P(D)=0.01$ (prior)
• $P(T|D)=0.95$ (likelihood)
• $P(T^c|D^c)=0.90$. This is the probability of a negative test given no disease. This implies the "false positive" rate is $P(T|D^c)=1-0.90=0.10$.
• $P(D^c)=1-P(D)=0.99$.

Now apply Bayes' Theorem:
$P(D|T)=\frac{P(T|D)\cdot P(D)}{P(T)}$
First, find the evidence $P(T)$ using the Law of Total Probability:
$P(T)=P(T|D)P(D)+P(T|D^c)P(D^c)$
$P(T)=(0.95\cdot 0.01)+(0.10\cdot 0.99)=0.0095+0.099=0.1085$
Now, calculate the posterior:
$P(D|T)=\frac{0.0095}{0.1085}\approx 0.0876$

This is a surprising result! Even with a positive test, there is only an 8.76% chance the person actually has the disease. This is because the disease is very rare in the population (low prior), and the false positive rate contributes significantly to the total number of positive tests. This type of counter-intuitive result is why Bayes' theorem is so crucial.

Conclusion

Today, you've moved from simply counting outcomes to reasoning about their likelihood. You've learned the fundamental rules that govern all probabilities and have been introduced to two of the most important tools in a data scientist's toolkit: conditional probability and Bayes' theorem.

Key Takeaways:

• Axioms of Probability: Probabilities are non-negative, the total probability of the sample space is 1, and the probabilities of mutually exclusive events add up.
• General Addition Rule: $P(A\cup B)=P(A)+P(B)-P(A\cap B)$.
• Conditional Probability: The probability of an event given that another has occurred is $P(A|B)=\frac{P(A\cap B)}{P(B)}$. It reflects a shrinking of the sample space.
• Bayes' Theorem: A powerful formula, $P(A|B)=\frac{P(B|A)P(A)}{P(B)}$, for updating our initial beliefs (prior) with new data (evidence) to get an updated belief (posterior).

Preview of the Next Lesson:

So far, we've talked about probabilities of abstract "events." In the next lesson, we will make this more concrete by introducing the concept of a random variable. We will learn to differentiate between discrete and continuous random variables and their distributions (PMF/PDF). This will allow us to start modeling and analyzing numerical data from real-world scenarios.